RAG Chunking Strategies: Size, Overlap, Structure and Model Limits

1.Why chunking is an early quality lever

In a RAG data pipeline, documents are parsed, deduplicated and filtered, then split into chunks. Each chunk is embedded and indexed. The aim is chunks that are small and focused enough to fit in the LLM's context without bringing in distracting text. The retriever can only return chunks, so the way you split documents limits what the LLM ever sees. Databricks calls chunking one of the first layers of optimization in a RAG application.

The cookbook lists five factors to decide when you chunk:

- Strategy: how the text is divided, from sentence, paragraph or token-count splits to document-specific methods. - Size: small chunks pinpoint details but drop the surrounding context. Large chunks keep context but bring in noise. - Overlap: sharing some text between neighbouring chunks keeps continuity, so a fact that falls on a boundary isn't lost. - Semantic coherence: each chunk should hold related information and still make sense on its own. - Metadata: the source document name, section heading or product names help match queries to chunks.

Changing any one of these changes the result set the LLM receives.

Sources1

2.The four families of chunking strategy

Databricks groups chunking strategies into four families. Moving down the list, each one splits text along more meaningful boundaries and takes more effort to set up. Fixed-size splitting is quick to configure, but it cuts text at arbitrary points, so chunks are often not semantically coherent. Paragraph-based splitting uses the document's own paragraph breaks. Format-specific splitting uses structure such as Markdown or HTML headers. Semantic chunking analyses the content itself, for example with topic modeling, to find where the topic changes.

Sources1

3.Model limits that bound chunk size

Two models put limits on your chunk size.

The embedding model. Every embedding model has a maximum number of input tokens. If a chunk is longer, the model truncates it, and the text past the limit never reaches the vector. For example, bge-large-en-v1.5 accepts at most 512 tokens. A 1,000-token chunk would be embedded from only its first part, even though all of the text is stored.

The generation LLM. The retriever usually returns several chunks, and all of them have to fit in the LLM's context window together with the prompt. So the right chunk size also depends on how many chunks you plan to retrieve. The embedding model's token limit is a hard maximum. It does not tell you the best chunk size.

Sources123

4.Beyond chunk size: semantic context and parent-child chunks

The AI Search retrieval quality guide calls chunk size optimization an active area of research. It recommends that you not spend most of your effort on chunk size. Higher-impact work includes extracting metadata such as entities and topics for filtering, parsing documents well (for example with ai_parse_document), and adding semantic context to chunks. One way to add context is to put the document title, a summary and the section name at the start of each chunk, so the embedding has signal from the whole document.

# Prepend document summary to each chunk
chunk_with_context = f"""
Document: {doc_title}
Summary: {doc_summary}
Section: {section_name}
{chunk_content}
"""

The guide lists two advanced strategies that take more effort but can have a bigger impact. Semantic chunking uses embeddings to group sentences by similarity, so related ideas stay together. Parent-child chunking, also called small-to-big retrieval, sidesteps the trade-off between small and large chunks. You search small child chunks, which locate information precisely, and return the larger parent chunk, which gives the LLM context.

# Record child and parent chunks in your source table
for parent_chunk in create_chunks(doc, size=2048):  # Large for context
    for child_chunk in create_chunks(parent_chunk, size=512):  # Small for precision
        source_table.append({"text": child_chunk, "parent_text": parent_chunk})

# Search children, return parents
results = index.similarity_search(
    query_text="Is attention all you need?",
    num_results=10,
    columns=["text", " ? "]
)

Sources2

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Fixed-size token splitting is an acceptable default for a production RAG application.Why is that wrong?

   Fixed-size splitting is quick to set up, but it seldom produces semantically coherent chunks. The cookbook says it rarely works in production.

   Covered in The four families of chunking strategy

2. 2.If the embedding model supports 8192 tokens, chunks should be 8192 tokens long.Why is that wrong?

   The token limit is a hard maximum, not a target. Databricks still recommends smaller chunks so the LLM receives a wider variety of examples.

   Covered in Model limits that bound chunk size

3. 3.Tuning chunk size is the highest-impact way to improve retrieval.Why is that wrong?

   The retrieval quality guide recommends spending effort on metadata extraction, high-quality parsing and adding semantic context to chunks instead of over-tuning chunk size.

   Covered in Beyond chunk size: semantic context and parent-child chunks