RAG Chunking Strategies: Fixed-Size, Paragraph, Format-Specific and Semantic

1.Where chunking sits and what it decides

A RAG data pipeline turns raw documents into something a retriever can query. The Databricks unstructured-pipeline guide breaks it into these stages: choosing the corpus, preprocessing (parsing, enrichment, metadata extraction, deduplication, filtering), chunking, embedding, and indexing. Chunking runs once the text is clean and before anything is embedded. This is where you decide the unit of text that retrieval will hand to the LLM.

Chunks have two jobs. They must be small enough that several of them fit in the LLM's context together, and focused enough that a retrieved chunk isn't padded with off-topic text. The guide puts it as chunking "ensures that retrieved data fits in the LLM's context while minimizing the inclusion of distracting or irrelevant information." Everything downstream sees only what the chunks contain, so the guide calls chunking "one of the first layers of optimization in an RAG application."

The guide names five factors to weigh when you design chunking:

- Chunking strategy: how the text is divided (by sentence, paragraph, character or token count, or document-specific structure). - Chunk size: small chunks focus on specific details but lose surrounding context. Large chunks keep context but bring in irrelevant material and cost more to compute. - Overlap: repeating some text between adjacent chunks so that information sitting on a boundary isn't lost. - Semantic coherence: chunks that can stand on their own, built by following paragraphs, sections or topic boundaries. - Metadata: source document name, section heading or product names attached to each chunk to help match queries.

Chunking isn't a self-contained problem. The quality overview points out that retrieval quality depends on choices in both the pipeline and the chain, and the pipeline choices it lists include the chunking strategy.

Sources12

2.Four strategies, matched to document structure

The guide says outright that "There is no one-size-fits-all approach." The right method depends on your use case and on how your documents are structured. It groups the options into four families, from cheapest to most involved.

Fixed-size chunking splits text every N characters or tokens. It's fast to set up, but the cuts land wherever the count runs out, often mid-sentence or mid-table, so the chunks are rarely coherent. The guide's verdict: "This approach rarely works for production-grade applications."

Paragraph-based chunking follows the paragraph breaks the author already wrote. Paragraphs usually hold related information, so this preserves coherence at little cost.

Format-specific chunking uses the structure built into formats like Markdown or HTML. Headers or sections become the chunk boundaries, so a chunk lines up with a section a human would recognise.

Semantic chunking looks at the content to find where the topic changes, for example with topic modeling or by comparing sentence embeddings. The AI Search retrieval quality guide describes it as grouping sentences by similarity instead of by fixed size, which keeps related ideas together. It takes more effort, but the boundaries match the text's natural divisions.

Whichever strategy you pick, overlap is a separate setting. Repeating a little text between neighbouring chunks protects facts that would otherwise be cut in two. According to the guide, overlap can preserve continuity and context and improve retrieval results.

Sources13

3.Sizing chunks against embedding and LLM limits

Two limits cap chunk size. The first is the embedding model's maximum token count. The guide's example is bge-large-en-v1.5, which has a limit of 512 tokens, and chunks longer than that get truncated. The second is the LLM context window. The retrieval quality guide notes that retrieved chunks must fit inside it, and you usually retrieve several at a time, so the space is shared.

A large embedding limit doesn't mean your chunks should be that large. The AI Search example notebooks use a model that accepts 8,192 tokens, yet they still advise splitting into smaller chunks so the reasoning model gets a wider variety of examples. The OpenAI-embedding example splits on a fixed token count of 1,024:

def chunk_text(text):
    # Encode and then decode within the UDF
    tokens = encoding.encode(text)
    chunks = []
    while tokens:
        chunk_tokens = tokens[:max_chunk_tokens]
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        tokens = tokens[max_chunk_tokens:]
    return chunks

For experiments, the retrieval quality guide suggests testing 256 tokens for precise fact retrieval, 512 as a middle ground, and 1,024 for more context per chunk. Smaller chunks pinpoint information better but can lose context. Larger chunks keep context but make the relevant passage harder to locate.

The guide also warns against spending too long on size. It calls chunk size an active research area, then says: "Instead of over-optimizing chunk size, focus on:" extracting metadata (entities, topics, categories) for filtering, parsing documents cleanly, and adding semantic metadata such as summaries and section headers to chunks.

Sources143

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Fixed-size character or token splitting is a sound production default because it's simple and predictable.Why is that wrong?

   Arbitrary character or token counts usually produce chunks that aren't semantically coherent. The guide says this approach rarely works in production; prefer paragraph, format-specific or semantic boundaries.

   Covered in Four strategies, matched to document structure

2. 2.Bigger chunks are always safer because they keep more context.Why is that wrong?

   Chunks over the embedding model's token limit are truncated, and retrieved chunks must also fit in the LLM context window together. Bigger chunks keep context but make relevant information harder to pinpoint.

   Covered in Sizing chunks against embedding and LLM limits

3. 3.Tuning chunk size is the main lever for improving retrieval quality.Why is that wrong?

   The retrieval quality guide calls chunk size an active research area and points to metadata extraction, high-quality parsing and semantic metadata as more impactful.

   Covered in Sizing chunks against embedding and LLM limits