How Document Chunking Works in RAG
When a RAG system receives a large document, it usually doesn't send the entire document to the AI model at once. Instead, the document is broken into smaller pieces called chunks.
Introduction
When a RAG system receives a large document, it usually doesn't send the entire document to the AI model at once. Instead, the document is broken into smaller pieces called chunks.
Document chunking is the process of splitting large documents into smaller, meaningful sections so they can be embedded, stored in a vector database, retrieved during a search, and passed to an AI model as context.
A good chunking strategy can make a RAG system dramatically more useful. A bad one can cause missing context, irrelevant search results, and inaccurate answers.
Here's how it works.
Quick Answer
Document chunking is the process of splitting a large document into smaller pieces before storing it in a RAG system.
Instead of embedding an entire 100-page PDF as one vector, you might split it into hundreds of smaller chunks.
The typical workflow looks like this:
Document
↓
Extract Text
↓
Split Into Chunks
↓
Create Embeddings
↓
Store in Vector Database
↓
Retrieve Relevant Chunks
↓
Send Context to LLM
↓
Generate Answer
The goal isn't simply to create smaller pieces.
The goal is to create useful pieces of information that can be retrieved independently.
That's the part developers sometimes miss.
What Is Document Chunking?
Imagine you have a 200-page employee handbook.
It contains information about:
Leave policies
Working hours
Health insurance
Remote work
Salary policies
Security rules
Expense reimbursement
Travel policies
If you store the entire handbook as one vector, a search for:
"How many vacation days do employees get?"
has very little precision.
The embedding represents the document as a whole.
Instead, you can split the handbook into smaller sections:
Chunk 1
Employee Leave Policy
Annual leave entitlement...
Chunk 2
Sick Leave
Employees may take...
Chunk 3
Remote Work Policy
Employees can work remotely...
Chunk 4
Travel Reimbursement
Employees can claim...
Now the retrieval system can find the chunk specifically related to vacation or annual leave.
This is the basic idea behind chunking.
Why Does RAG Need Chunking?
RAG systems rely on retrieval.
The system first searches a knowledge base for information relevant to the user's question. It then gives the retrieved information to the language model.
If the stored information is too large or poorly organized, retrieval becomes harder.
For example:
User Question
│
▼
"How much annual leave do I get?"
│
▼
Search Knowledge Base
│
▼
Relevant Chunk
"Employees receive 20 days
of annual leave per year."
│
▼
LLM
│
▼
Answer
"You receive 20 days..."
The chunk acts as the bridge between the user's question and the original document.
This is why chunking directly affects retrieval quality.
If you're new to the broader RAG architecture, it's worth reading What Is RAG? Retrieval-Augmented Generation Explained before going deeper into chunking.
How Document Chunking Works
At a high level, chunking has four steps.
Step 1: Load the Document
First, your application reads a source document.
The source could be:
PDF
HTML page
Markdown file
Word document
Text file
Database record
API response
Web page
For example:
company-handbook.pdf
is loaded into the application.
Step 2: Extract the Text
The application extracts readable text from the document.
For a PDF, the extracted content might look like:
Company Leave Policy
Employees receive 20 days of annual leave.
Leave requests should be submitted through
the employee portal.
Sick leave is available for eligible employees...
At this point, the application has text but hasn't created chunks yet.
Step 3: Split the Text
The extracted text is divided into smaller pieces.
For example:
Original Document
│
▼
────────────────────
Company Leave Policy
────────────────────
│
├── Chunk 1
├── Chunk 2
├── Chunk 3
└── Chunk 4
Each chunk should contain enough information to be useful on its own.
Step 4: Create Embeddings
Once the text has been divided, each chunk can be converted into an embedding.
For example:
Chunk 1
"Employees receive 20 days of annual leave."
↓
Embedding Model
↓
[0.12, -0.44, 0.87, 0.21, ...]
These embeddings can then be stored in a vector database.
If you want to understand the embedding step in more detail, see What Are Embeddings ?
A Simple Chunking Example
Suppose we start with this document:
Python is a popular programming language.
It is commonly used for web development,
automation, data science, and machine learning.
Python has a large ecosystem of libraries.
Developers can install packages using pip.
We could split it into four chunks:
Chunk 1:
Python is a popular programming language.
Chunk 2:
It is commonly used for web development,
automation, data science, and machine learning.
Chunk 3:
Python has a large ecosystem of libraries.
Chunk 4:
Developers can install packages using pip.
This looks simple.
But real documents aren't always this clean.
A technical PDF might contain:
Chapter
↓
Section
↓
Subsection
↓
Paragraph
↓
Sentence
↓
Table
↓
Code Block
A good chunking strategy needs to understand this structure.
Chunk Size
Chunk size determines how much text each chunk contains.
You might define chunk size using:
Characters
Words
Sentences
Tokens
Token-based chunking is especially common in LLM applications because language models process text using tokens.
For example:
Document
↓
500-token chunks
↓
Embedding
↓
Vector Database
A chunk that's too small can lose important context.
A chunk that's too large can contain too much unrelated information.
So what's the ideal size?
There isn't one universal answer.
It depends on the data and the retrieval task.
Small Chunks vs Large Chunks
Consider these two approaches.
Small Chunk
Employees receive 20 days
of annual leave.
This is highly focused.
But it might lack important context.
Maybe the next paragraph says:
Employees must complete their
probation period before becoming
eligible for annual leave.
The first chunk alone could be misleading.
Large Chunk
Leave Policy
Employees receive 20 days of annual
leave. Employees must complete their
probation period before becoming
eligible...
Requests must be submitted through
the employee portal...
This contains more context.
But now the chunk may contain information unrelated to many searches.
That's the trade-off.
Chunk Overlap
Chunk overlap means repeating some text between consecutive chunks.
Suppose we have:
Chunk 1
Python is used for web development.
It is also popular for automation.
Python has a large ecosystem.
Chunk 2
Python has a large ecosystem.
Developers can install packages using pip.
The repeated sentence:
Python has a large ecosystem.
is the overlap.
Without overlap, important information can be split between two chunks.
Why Chunk Overlap Matters
Consider this sentence:
The API requires OAuth authentication.
Access tokens expire after 60 minutes.
If the split happens here:
Chunk 1:
The API requires OAuth authentication.
Chunk 2:
Access tokens expire after 60 minutes.
A search for:
"How long do API access tokens last?"
can still find Chunk 2.
But some questions need both pieces:
"What authentication method does the API use and how long do tokens last?"
Now the relationship between the two sentences matters.
Overlap can help preserve that relationship.
For example:
Chunk 1:
The API requires OAuth authentication.
Access tokens expire after 60 minutes.
Chunk 2:
Access tokens expire after 60 minutes.
Refresh tokens can be used to obtain...
The repeated context gives the retrieval system a better chance of finding complete information.
Don't overdo overlap, though.
Too much overlap creates duplicate content and increases storage and embedding costs.
Common Chunking Strategies
There are several ways to split documents.
The most common approaches include:
Chunking Strategies
│
├── Fixed-Size
├── Sentence-Based
├── Paragraph-Based
├── Recursive
├── Structure-Aware
└── Semantic
Each strategy solves a slightly different problem.
Fixed-Size Chunking
Fixed-size chunking splits text into pieces with a predetermined size.
For example:
500 tokens per chunk
or:
1,000 characters per chunk
The algorithm doesn't care much about the meaning of the text.
It simply divides the content based on the configured limit.
Example:
text = "A very long document..."
chunk_size = 500
chunks = [
text[0:500],
text[500:1000],
text[1000:1500]
]
In real applications, token-aware libraries are usually preferable to manually slicing strings like this.
Advantages
Fixed-size chunking is:
Simple
Fast
Easy to implement
Predictable
Useful for experimentation
Disadvantages
It can split content in awkward places.
For example:
Chunk 1:
The authentication process requires
Chunk 2:
OAuth 2.0 with a refresh token...
The meaning is divided across chunks.
That's not ideal.
Sentence-Based Chunking
Sentence-based chunking splits documents according to sentence boundaries.
For example:
Sentence 1
Sentence 2
Sentence 3
Sentence 4
Instead of cutting text at an arbitrary character count, the system tries to preserve complete sentences.
Example:
import re
text = """
Python is easy to learn.
It has a large ecosystem.
It is widely used in AI.
"""
sentences = re.split(r'(?<=[.!?])\s+', text.strip())
for sentence in sentences:
print(sentence)
Output:
Python is easy to learn.
It has a large ecosystem.
It is widely used in AI.
This approach preserves sentence boundaries.
However, individual sentences can still be too small for meaningful retrieval.
You can solve that by grouping multiple sentences together.
Paragraph-Based Chunking
Paragraph-based chunking uses existing document paragraphs as boundaries.
For example:
Paragraph 1 → Chunk 1
Paragraph 2 → Chunk 2
Paragraph 3 → Chunk 3
This often works surprisingly well for clean documentation.
Why?
Because writers already grouped related ideas into paragraphs.
For example:
OAuth authentication is required for all API requests.
Clients must include a valid access token...
That paragraph may already represent a coherent unit of information.
The problem is that paragraph sizes can vary significantly.
One paragraph might contain 40 words.
Another might contain 800.
So production systems often combine paragraph boundaries with a maximum chunk size.
Recursive Chunking
Recursive chunking is one of the most useful general-purpose approaches.
Instead of immediately cutting text at a fixed size, the algorithm tries several separators.
For example:
Document
↓
Paragraph
↓
Sentence
↓
Word
↓
Character
The idea is simple.
Try to preserve the largest meaningful structure possible.
For example:
1. Split by paragraphs
2. If a piece is too large
3. Split that piece by sentences
4. If it's still too large
5. Split by smaller units
This produces chunks that are usually more coherent than simple fixed-size splitting.
A simplified implementation might look like this:
def recursive_split(text, max_length):
if len(text) <= max_length:
return [text]
paragraphs = text.split("\n\n")
chunks = []
for paragraph in paragraphs:
if len(paragraph) <= max_length:
chunks.append(paragraph)
else:
sentences = paragraph.split(". ")
current = ""
for sentence in sentences:
if len(current) + len(sentence) <= max_length:
current += sentence + ". "
else:
chunks.append(current.strip())
current = sentence + ". "
if current:
chunks.append(current.strip())
return chunks
This is intentionally simplified.
Production implementations should account for punctuation, Unicode, token counts, overlap, metadata, and document structure.
Structure-Aware Chunking
Structure-aware chunking looks at the document's actual organization.
This is especially useful for:
Technical documentation
Markdown
HTML
Legal documents
Research papers
Product documentation
API references
Consider a Markdown document:
# Authentication
## API Keys
API keys authenticate requests.
## OAuth
OAuth provides delegated access.
## Refresh Tokens
Refresh tokens can obtain new access tokens.
A structure-aware chunker can preserve the headings.
Instead of storing:
OAuth provides delegated access.
you might store:
Authentication > OAuth
OAuth provides delegated access.
That's much more useful.
The heading provides additional context.
Metadata Makes Chunks Better
You can also attach metadata to each chunk.
For example:
chunk = {
"text": "OAuth provides delegated access.",
"section": "OAuth",
"parent_section": "Authentication",
"source": "api-guide.md",
"page": 12
}
Now retrieval returns not just text, but context about where that text came from.
Metadata can help with:
Filtering
Citations
Debugging
Ranking
Access control
Source attribution
This becomes especially important in production RAG systems.
Semantic Chunking
Semantic chunking takes a different approach.
Instead of asking:
"How many characters should each chunk contain?"
it asks:
"Which sentences belong together conceptually?"
For example:
Sentence 1:
Python is commonly used for machine learning.
Sentence 2:
Libraries such as PyTorch and TensorFlow support
deep learning workloads.
Sentence 3:
The Python package manager is pip.
Sentence 4:
Packages can be installed from PyPI.
A semantic chunker might group:
Chunk 1
Python + machine learning + PyTorch + TensorFlow
and:
Chunk 2
pip + PyPI + package installation
The boundaries are based more on meaning than position.
Semantic chunking can improve retrieval for complex datasets.
But there's a trade-off.
It usually requires more computation and can be harder to tune than basic chunking strategies.
Chunking in a RAG Pipeline
Now let's connect everything.
A typical RAG ingestion pipeline looks like this:
DOCUMENT
│
▼
Text Extraction
│
▼
Chunking
│
▼
Add Metadata + Overlap
│
▼
Embeddings
│
▼
Vector Database
│
│
User asks question
│
▼
Query Embedding
│
▼
Retrieval
│
▼
Relevant Chunks
│
▼
LLM
│
▼
Answer
Notice something important.
Chunking happens before embeddings.
You don't normally embed the entire document and then decide how to split it later.
The usual process is:
Document
→ Chunks
→ Embeddings
→ Vector Store
This is why chunking is part of the ingestion pipeline.
Python Example
Let's build a small chunking example without using a framework.
Suppose we have:
document = """
Python is a popular programming language.
It is commonly used for web development,
automation, data science, and machine learning.
Python has a large ecosystem of libraries.
Developers can install packages using pip.
"""
We can create a simple paragraph-based chunker:
def chunk_document(text, max_length=120):
paragraphs = [
paragraph.strip()
for paragraph in text.split("\n\n")
if paragraph.strip()
]
chunks = []
current_chunk = ""
for paragraph in paragraphs:
if len(current_chunk) + len(paragraph) <= max_length:
current_chunk += paragraph + "\n\n"
else:
if current_chunk:
chunks.append(current_chunk.strip())
current_chunk = paragraph + "\n\n"
if current_chunk:
chunks.append(current_chunk.strip())
return chunks
document = """
Python is a popular programming language.
It is commonly used for web development,
automation, data science, and machine learning.
Python has a large ecosystem of libraries.
Developers can install packages using pip.
"""
chunks = chunk_document(document)
for index, chunk in enumerate(chunks, start=1):
print(f"Chunk {index}:")
print(chunk)
print()
The function first separates paragraphs.
It then combines paragraphs until adding another one would exceed the configured size.
This is a basic example, but the same concept exists inside many RAG frameworks.
How Chunk Size Affects Retrieval
Chunk size can influence retrieval in several ways.
Consider a question:
"How do I reset my password?"
Very Small Chunks
You might retrieve:
Click Settings.
But the actual password-reset instructions could be in another chunk.
The answer lacks context.
Very Large Chunks
You might retrieve:
Account Settings
Profile Settings
Password Management
Two-Factor Authentication
Email Notifications
Security Preferences
...
Now the relevant password information is buried inside a large amount of unrelated text.
Better Chunk
Password Reset
To reset your password, open Account Settings,
select Security, and click Reset Password.
That's exactly the kind of information retrieval systems want.
The chunk is focused but still understandable.
Common Chunking Mistakes
One mistake I see often is treating chunking as a simple preprocessing task.
It isn't.
Chunking is part of your retrieval design.
Here are some common problems.
1. Using the Same Chunk Size for Everything
A legal contract, a code repository, and a FAQ page don't have the same structure.
A single configuration may work for one and perform badly for another.
2. Making Chunks Too Small
Tiny chunks may contain incomplete ideas.
For example:
The function returns
is not useful without the rest of the sentence.
3. Making Chunks Too Large
Large chunks contain more information, but retrieval becomes less focused.
The model may receive lots of irrelevant context.
4. Ignoring Document Structure
If your document contains:
Chapter
Section
Subsection
don't throw all of that structure away during ingestion.
Preserve it as metadata where possible.
5. Forgetting Overlap
Hard boundaries can separate related sentences.
A small amount of overlap can preserve continuity.
6. Ignoring Metadata
A chunk without its source information can become difficult to trace later.
Store useful metadata such as:
document_id
source
page
section
heading
chunk_id
7. Optimizing Only for Storage
Fewer chunks aren't automatically better.
You aren't trying to minimize the number of vectors.
You're trying to maximize the quality of retrieved context.
That's a very different goal.
How to Choose the Right Chunk Size
There's no magic number.
Start with a reasonable value and test it against real questions.
For example:
Experiment A
Small chunks + low overlap
Experiment B
Medium chunks + moderate overlap
Experiment C
Large chunks + moderate overlap
Then evaluate retrieval.
Ask:
Did the correct chunk appear?
Was enough context included?
Was irrelevant information retrieved?
Were related facts split across chunks?
Did the answer contain enough evidence?
How many chunks were needed?
This is much better than choosing a chunk size because someone online recommended it.
Chunking Strategy by Data Type
Different data often benefits from different approaches.
FAQs
Paragraph or question-answer based chunking usually works well.
Question
+
Answer
Keep them together.
Technical Documentation
Structure-aware or recursive chunking is usually a good starting point.
Preserve:
Title
Heading
Subheading
Code
Explanation
PDFs
PDFs require extra care because visual structure doesn't always survive text extraction.
Tables, columns, headers, and footers can produce messy text.
Source Code
Code should generally be chunked around logical structures such as:
Class
Function
Method
Module
Splitting code at arbitrary character counts can destroy useful context.
Legal Documents
Preserve sections, clauses, article numbers, and headings.
A chunk should retain enough context to identify the clause it belongs to.
A Better Mental Model
Think about chunks as retrievable knowledge units.
Don't ask:
"How can I split this document into 500-token pieces?"
Ask:
"What is the smallest piece of information that still makes sense when retrieved by itself?"
That's a much better starting point.
For example:
Bad chunk:
"before submitting the request"
Good chunk:
"Employees must submit leave requests
through the employee portal before taking leave."
The second chunk has a clear meaning even when retrieved independently.
That's what you want.
Chunking and Embeddings
Chunking and embeddings are closely connected.
The embedding model converts each chunk into a numerical representation.
For example:
Chunk
"Employees receive 20 days of annual leave."
↓
Embedding Model
↓
[0.21, -0.08, 0.72, 0.41, ...]
Later, a user asks:
"How many vacation days do I get?"
The query is also converted into an embedding.
The RAG system compares the query representation with stored chunk representations.
If you want to understand this process more deeply, the previous article on Vector Databases Explained covers where those embeddings are stored and searched.
Chunking and Semantic Search
Chunking also affects semantic search.
Suppose a document contains:
The company supports remote work.
Employees can work remotely up to three days per week.
Managers must approve remote schedules.
A semantic search query such as:
"How often can I work from home?"
should ideally retrieve the chunk containing the remote-work policy.
If the chunking process separates:
Employees can work remotely
from:
up to three days per week.
the retrieval result may lose important meaning.
This is one reason semantic chunking and structure-aware chunking can be useful for complex documents.
For more background, see Semantic Search vs Keyword Search.
How Chunking Affects RAG Quality
A RAG system can fail even when the LLM is excellent.
Consider this pipeline:
Bad Chunking
↓
Bad Retrieval
↓
Bad Context
↓
Good LLM
↓
Bad Answer
The language model can only work with the context it receives.
If the retrieval system returns the wrong information, the model can't magically retrieve the missing facts from your private knowledge base.
This is why improving the LLM isn't always the solution.
Sometimes you need to go back to the ingestion pipeline.
Practical Chunking Workflow
When building a new RAG application, I recommend starting simple.
Use this workflow:
1. Understand your documents
↓
2. Preserve document structure
↓
3. Start with recursive chunking
↓
4. Add moderate overlap
↓
5. Store useful metadata
↓
6. Create embeddings
↓
7. Test retrieval with real questions
↓
8. Adjust chunk size
↓
9. Compare retrieval quality
↓
10. Keep the simplest strategy that works
Don't start with an unnecessarily complicated semantic chunking system.
Get the fundamentals working first.
Then measure.
Best Practices for Document Chunking
Here are the practices worth keeping in mind.
1. Preserve Meaning
A chunk should represent a coherent idea.
2. Preserve Structure
Keep headings, sections, page numbers, and source information when possible.
3. Use Overlap Carefully
Overlap can preserve context, but excessive overlap creates duplication.
4. Test Multiple Sizes
Try different chunk sizes against real queries.
5. Keep Metadata
Metadata makes retrieval, filtering, debugging, and citations easier.
6. Chunk According to the Data
Don't use the same strategy blindly for PDFs, code, FAQs, and legal documents.
7. Evaluate Retrieval Separately
Before blaming the LLM, check whether the correct chunks were retrieved.
8. Keep an Eye on Cost
More chunks mean more embeddings and potentially more storage.
9. Inspect Real Chunks
Print and manually inspect your chunks during development.
This catches surprisingly many problems.
10. Optimize for Retrieval
The goal is not perfect-looking chunks.
The goal is useful retrieved context.
Common Questions Developers Ask
Does every RAG system need document chunking?
For most document-based RAG systems, yes.
If your knowledge sources are already small, atomic records, you may not need traditional document chunking.
A database row or FAQ entry may already be an appropriate retrieval unit.
Is larger chunk size always better?
No.
Larger chunks provide more context but can reduce retrieval precision.
Smaller chunks improve focus but can lose surrounding context.
The right balance depends on your data and queries.
What is chunk overlap?
Chunk overlap repeats some content between neighboring chunks.
It helps prevent important context from being lost at chunk boundaries.
How much overlap should I use?
There's no universal value.
A modest overlap is a reasonable starting point, then evaluate it using real queries.
More overlap isn't automatically better.
Should I chunk PDFs differently?
Often, yes.
PDFs can contain tables, columns, headers, footers, images, and other layout elements that complicate text extraction.
Good PDF chunking starts with good document parsing.
Should headings be included in chunks?
Usually, yes.
Headings provide context and can make retrieved chunks easier for both retrieval systems and language models to understand.
Should code be chunked like normal text?
Usually not.
Code is better divided around logical structures such as functions, classes, methods, and modules.
Does chunking affect embeddings?
Yes.
The embedding represents the chunk you provide to the embedding model.
If the chunk contains unrelated information, the resulting representation may be less focused.
Can bad chunking cause hallucinations?
It can contribute to poor answers.
If retrieval returns incomplete or irrelevant context, the model may lack the evidence required to answer correctly.
Chunking isn't the only cause of hallucinations, but it can have a major effect on retrieval quality.
Should I use semantic chunking immediately?
Not necessarily.
Start with a simpler approach such as recursive or structure-aware chunking.
Measure retrieval quality.
Move to more sophisticated approaches when your data actually needs them.
Key Takeaways
Document chunking is one of the most important steps in a RAG ingestion pipeline.
Remember these points:
Chunking splits large documents into retrievable pieces.
Chunk size affects both context and retrieval precision.
Overlap helps preserve information across boundaries.
Different document types need different strategies.
Recursive chunking is a useful general starting point.
Structure-aware chunking can preserve headings and document hierarchy.
Semantic chunking groups content based on meaning.
Metadata makes chunks easier to retrieve, filter, debug, and cite.
More chunks aren't automatically better.
The best chunking strategy is the one that produces useful retrieval results for your actual data.
The biggest lesson is simple:
Don't chunk documents just to make them smaller. Chunk them so the information can be retrieved effectively.
Conclusion
Chunking looks like a small preprocessing step, but it has a huge influence on a RAG system.
Your vector database can be fast. Your embedding model can be excellent. Your LLM can be highly capable.
If the document was split badly, retrieval can still fail.
That's why chunking should be treated as part of your RAG architecture rather than an afterthought.
Start with a simple strategy. Preserve document structure. Use reasonable overlap. Store metadata. Then test the system with real questions.
Once you understand how document chunking works, the next step is to explore different chunking strategies for different types of data.
That's where things get more interesting.
Continue Learning
You're currently at Article 5 in the RAG series.
Previous Articles
1. What Is RAG? Retrieval-Augmented Generation Explained
Learn the basic RAG architecture and understand how retrieval and generation work together.
Understand how text is converted into numerical representations for semantic retrieval.
Learn how vector databases store and search embeddings.
4. Semantic Search vs Keyword Search
Understand the difference between exact-term matching and meaning-based retrieval.
Next Articles
6. Chunking Strategies for RAG
Explore fixed-size, recursive, semantic, structure-aware, and specialized chunking strategies.
7. Build Your First RAG Application with Python
Put the concepts together and build a working RAG application.
8. RAG with OpenAI API
Learn how to build RAG applications using the OpenAI API.
9. RAG with LangChain
See how LangChain can simplify document loading, chunking, retrieval, and generation.
10. RAG with LlamaIndex
Explore another popular framework for building document-based RAG applications.
Follow TheTechLane
Keep learning AI, development, and modern developer technologies with TheTechLane.
Learn • Build • Share
Follow @thetechlanehq for more developer guides, tutorials, and practical technology content.