Chunking Strategies for Document Structure and Model Limits

1.Where chunking sits, and what you are deciding

A RAG application over unstructured content needs a data pipeline. The pipeline parses raw files, cleans them, cuts them into chunks, embeds each chunk and indexes the vectors. Chunking comes after the text has been extracted and cleaned, and before anything is embedded. Because of where it sits, every later stage works on whatever chunks you produce. The Databricks guidance calls chunking one of the first layers of optimization in a RAG application, because it directly controls what the LLM is given at answer time.

"Choose a chunk size" is only one part of a chunking decision. The guidance lists five factors. Strategy is the method used to cut the text, for example by sentences, paragraphs, character or token counts, or document-specific rules. Chunk size is a trade-off: smaller chunks focus on specific details but lose surrounding context, and larger chunks keep context but can pull in irrelevant text and cost more to process. Overlap between neighbouring chunks keeps information from being lost at a boundary. Semantic coherence means each chunk should hold related information and still make sense on its own. Metadata, such as the source document name, section heading or product name, gives retrieval queries more to match against.

Sources1

2.Matching the strategy to the document's structure

The Databricks guidance does not name a single best method. It says the right chunk size and method depend on the use case and on the data. It groups the options into four families, and each one ties a kind of document structure to a kind of boundary. Fixed-size splitting ignores structure entirely: it is quick to set up, but it usually does not produce coherent chunks. Paragraph-based splitting uses the paragraph breaks the author already wrote. Format-specific splitting uses explicit markup such as headers. Semantic chunking looks at the content to find where the topic changes, which helps with prose that has little visible structure.

Sources1

3.Sizing chunks against the embedding model and the LLM

Two model limits cap chunk size. The first is the embedding model's maximum token limit. The guidance gives bge-large-en-v1.5 as an example, with a limit of 512 tokens. Any chunk longer than the limit is cut off before it is embedded. The second is the LLM's context window. You usually retrieve several chunks for each query, and all of them have to fit into the prompt together. So the limit that matters is chunk size multiplied by the number of chunks you retrieve, not the size of one chunk.

Fitting under the embedding model's limit is necessary but not enough. The AI Search embedding examples point out that the GTE model, and the OpenAI model in the external example, support up to 8192 tokens. Databricks still recommends splitting data into smaller chunks, so that the model generating the answer receives a wider range of examples. The OpenAI example caps chunks at 1024 tokens.

max_chunk_tokens =  ? 
encoding = tiktoken.get_encoding("cl100k_base")


def chunk_text(text):
    # Encode and then decode within the UDF
    tokens = encoding.encode(text)
    chunks = []
    while tokens:
        chunk_tokens = tokens[:max_chunk_tokens]
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        tokens = tokens[max_chunk_tokens:]
    return chunks

For choosing a size within those limits, the AI Search retrieval quality guide suggests three starting points to test. It also says chunk size optimization is still an active area of research, so treat these as experiments rather than answers.

# Common configurations to test
small_chunks = 256   # Better for precise fact retrieval
medium_chunks = 512  # Balanced approach
large_chunks = 1024  # More context per chunk

Sources12

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Fixed-size chunking by character or token count is a sound default for a production RAG system.Why is that wrong?

   Fixed-size splitting is quick to set up, but it typically does not produce semantically coherent chunks. The guidance says it rarely works in production. Use the document's paragraphs, markup or topics to set boundaries instead.

   Covered in Matching the strategy to the document's structure

2. 2.If the embedding model accepts 8192 tokens, chunks should be close to 8192 tokens.Why is that wrong?

   The limit is a ceiling, not a target. Databricks recommends smaller chunks so the reasoning model receives a wider range of examples.

   Covered in Sizing chunks against the embedding model and the LLM

3. 3.A chunk longer than the embedding model's limit causes an error, so you will notice it.Why is that wrong?

   Oversized chunks are truncated without any error, so the text past the limit never reaches the embedding.

   Covered in Sizing chunks against the embedding model and the LLM