What you will be able to do
- Decide whether a knowledge base needs a RAG pipeline at all, or fits in the prompt
- Explain how chunk size, boundary and overlap affect retrieval, and why chunks lose context
- Select an embedding model and embed documents and queries with the correct input type
- Design a hybrid index that combines semantic embeddings with BM25 lexical search
Key concept
Chunk context loss — Splitting documents into small chunks makes retrieval efficient, but each chunk is embedded and indexed on its own, without the surrounding document. A chunk that depends on that missing context (which company, which quarter, which file) becomes hard to retrieve and hard to use.
1.First decide whether you need RAG
Retrieval-Augmented Generation (RAG) lets Claude answer from your own corpus: support articles, internal documents, filings, codebases. Before you design chunking or indexing, check how big the knowledge base is. Anthropic's Contextual Retrieval post gives a simple threshold. If the knowledge base is under 200,000 tokens (about 500 pages), put all of it in the prompt. Prompt caching cuts latency by more than 2x and cost by up to 90% for prompts that repeat across calls. A retrieval pipeline only earns its complexity once the corpus no longer fits.
For larger corpora, the standard pipeline has an offline half and a runtime half. Offline, you split the corpus into chunks, turn each chunk into an embedding vector, and store the vectors in a database that supports similarity search. At runtime, you search that database with the user's query and add the most relevant chunks to the prompt sent to Claude. The embedding step makes this work: an embedding model maps text to numbers so that passages with similar meaning end up close together. Every later design choice in this lesson (chunk boundaries, model choice, a second lexical index) is about making that runtime search return the right chunks.
Checkpoint 1 of 5· Check yourself
In a standard RAG pipeline for a corpus too large for the context window, what happens at query time?
At runtime the vector database ranks stored chunks by semantic similarity to the query. The top chunks go into the prompt for the generative model.
“the vector database is used to find the most relevant chunks based on semantic similarity to the query”Source: www.anthropic.com
2.Chunking: size, boundaries and the context problem
Chunks are usually no more than a few hundred tokens. Within that range there is no fixed setting to memorise. Anthropic lists chunk size, chunk boundary and chunk overlap as the three levers that affect retrieval performance, and the sources leave the values to experimentation on your own data. Two reference points: the cookbook's baseline ('Naive RAG') chunks documents by heading, with each chunk holding only the content under one subheading, and Anthropic's cost example assumes 800-token chunks from 8k-token documents.
Whatever size you pick, the chunk has to fit inside the embedding model's input limit. The Voyage text models listed in Claude's documentation accept 32,000 tokens, except voyage-law-2 at 16,000. The cookbook warns that if you prepend extra text to chunks and retrieval gets worse, the chunks may be getting truncated. In that case, use an embedding model with a larger context window.
The harder problem is that small chunks lose their surroundings. Take SEC filings and the question 'What was the revenue growth for ACME Corp in Q2 2023?'. The relevant chunk might say only: 'The company's revenue grew by 3% over the previous quarter.' Nothing in that chunk names the company or the period, so it is hard to retrieve for this query and hard for Claude to use once retrieved. You can fix this at indexing time. One option is to have Claude write chunk-specific context and prepend it to each chunk (Contextual Retrieval). Another is a contextualized chunk embedding model such as voyage-context-3 or voyage-context-4, which encode each chunk with the full document in view.
Checkpoint 2 of 5· Check yourself
A chunk reading 'The company's revenue grew by 3% over the previous quarter' is rarely retrieved for 'ACME Corp revenue growth Q2 2023'. What is the root cause?
The chunk was embedded in isolation, so the identifying information (company, quarter) that lives elsewhere in the document is missing from both the vector and the text.
“traditional RAG solutions remove context when encoding information”Source: www.anthropic.com
3.Choosing and calling the embedding model
Anthropic does not offer its own embedding model, so the embedding step always uses a third-party provider. The documentation names Voyage AI as one provider and tells you to assess several. It gives three selection factors. The first is the size and domain relevance of the model's training data. The second is inference performance, meaning lookup speed and end-to-end latency, which matters most in large production deployments. The third is customization, such as continued training on private data, which helps with unusual vocabularies. In Anthropic's Contextual Retrieval experiments, Gemini and Voyage embeddings were particularly effective.
| Model | Context length | Fits when |
|---|---|---|
| voyage-4-large | 32,000 | You want the best general-purpose and multilingual retrieval quality |
| voyage-4-lite | 32,000 | Latency and cost dominate |
| voyage-code-3 | 32,000 | The corpus is code |
| voyage-finance-2 | 32,000 | Finance retrieval and RAG |
| voyage-law-2 | 16,000 | Legal and long-context retrieval; note the smaller input limit |
| voyage-context-4 | 120,000 | You want chunk vectors that capture full document context (called with contextualized_embed()) |
Indexing and querying use the same model but different input types. Corpus chunks are embedded as documents when you build the index:
# Embed the documents
doc_embds = vo.embed(documents, model="voyage-4", input_type="document").embeddingsAt query time the user's question is embedded with the query input type and compared against the stored vectors. Voyage vectors are normalized to length 1, so a dot product gives the same result as cosine similarity, and the highest score is the nearest chunk.
Checkpoint 3 of 5· Fill the gap
Which input type completes the query-time embedding call?
query_embd = vo.embed([query], model="voyage-4", input_type=" ? ").embeddings[0]Documents are embedded with input_type="document" and the user's question with input_type="query". Mixing them up gives inconsistent vectors.
Source: platform.claude.com4.Hybrid indexing: embeddings plus BM25
An embedding index finds chunks by meaning, so it handles paraphrases well. It can miss exact matches. Suppose a user searches a technical support database for 'Error code TS-999'. An embedding model might return content about error codes in general and miss the one document that contains that exact string. BM25 (Best Matching 25) covers that gap. It is a lexical ranking function built on TF-IDF that also accounts for document length and applies a saturation function to term frequency, so common words do not dominate the results. It works best on queries with unique identifiers or technical terms.
A hybrid index therefore keeps two representations of every chunk: a TF-IDF/BM25 index for exact terms and an embedding for semantic similarity. At query time both are searched, and their result lists are merged with rank fusion and deduplicated before the top-K chunks go to Claude. This balances precise term matching against broader semantic understanding. It also scales to corpora far larger than any single prompt.
Checkpoint 4 of 5· Put it in order
Put the steps of a hybrid embeddings + BM25 RAG pipeline in order
- 1.Break the corpus into chunks of no more than a few hundred tokens
- 2.Add the top-K chunks to the prompt to generate the response
- 3.Create TF-IDF encodings and semantic embeddings for the chunks
- 4.Use BM25 to find top chunks based on exact matches
- 5.Combine and deduplicate the two result sets using rank fusion
- 6.Use embeddings to find top chunks based on semantic similarity
Chunking and dual indexing happen offline. At query time both retrievers run, rank fusion merges their results, and only the fused top-K reach the model.
“Combine and deduplicate results from (3) and (4) using rank fusion techniques”Source: www.anthropic.com
Checkpoint 5 of 5· Exam question
A search team wants their lexical index to keep matching exact product codes and error strings verbatim, while also benefiting from the same document-level grounding they just added to their semantic embeddings. They plan to keep both a BM25 index and a vector index in the pipeline. What should they do to align the two indexes?
Correct answer: A — Prepend the same chunk-specific contextual text used for embeddings to each chunk before it is tokenized for the BM25 index.
- A. Correct. This is Contextual BM25: indexing the same contextualized chunk text used for embeddings lets the lexical index capture both exact matches and the semantic relationships from the prepended context, aligning it with the contextual embeddings.
- B. Incorrect. Replacing BM25 with another embedding index removes the exact-match lexical matching (product codes, error strings) that the team explicitly wants to preserve.
- C. Incorrect. Reducing term-frequency weighting suppresses the lexical precision the team relies on for verbatim matches and does nothing to align the index with document-level context.
- D. Incorrect. Indexing only titles and headings discards the chunk body text needed for exact lexical matches on codes and error strings, worsening rather than improving alignment.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Any knowledge base you want Claude to use needs a chunk-embed-retrieve pipeline.Why is that wrong?
Below about 200,000 tokens (around 500 pages), Anthropic suggests putting the whole knowledge base in the prompt, with prompt caching to keep it fast and cheap. RAG is for corpora that do not fit.
Covered in First decide whether you need RAG
2.A good enough embedding model makes a lexical index redundant.Why is that wrong?
Embeddings capture meaning but can miss exact strings such as error codes and identifiers. BM25 catches those, which is why the recommended design combines both.
Covered in Hybrid indexing: embeddings plus BM25
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Text embeddings are numerical representations of text that enable measuring semantic similarity.”
↩︎ First decide whether you need RAG“produce chunk-level vectors that capture full document context without manual metadata augmentation”
↩︎ Chunking: size, boundaries and the context problem“Anthropic does not offer its own embedding model.”
↩︎ Choosing and calling the embedding model“Inference performance: embedding lookup speed and end-to-end latency. This is a particularly important consideration for large scale production deployments”
↩︎ Choosing and calling the embedding model“are used for embedding the document and query, respectively”
↩︎ Choosing and calling the embedding model“The embeddings allow you to do semantic search / retrieval in the vector space.”
↩︎ Hybrid indexing: embeddings plus BM25 - 2.https://www.anthropic.com/engineering/contextual-retrievalSecondary source
“Store these embeddings in a vector database that allows for searching by semantic similarity.”
↩︎ First decide whether you need RAG“The choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance”
↩︎ Chunking: size, boundaries and the context problem“smaller chunks of text, usually no more than a few hundred tokens”
↩︎ Chunking: size, boundaries and the context problem“We found Gemini and Voyage embeddings to be particularly effective.”
↩︎ Choosing and calling the embedding model“BM25 refines this by considering document length and applying a saturation function to term frequency”
↩︎ Hybrid indexing: embeddings plus BM25“particularly effective for queries that include unique identifiers or technical terms”
↩︎ Hybrid indexing: embeddings plus BM25“traditional RAG solutions remove context when encoding information”
↩︎ Key concept“you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG”
↩︎ Exam trap 1“While embedding models excel at capturing semantic relationships, they can miss crucial exact matches.”
↩︎ Exam trap 2“you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG”
↩︎ Prediction“the vector database is used to find the most relevant chunks based on semantic similarity to the query”
↩︎ Checkpoint“traditional RAG solutions remove context when encoding information”
↩︎ Checkpoint“Combine and deduplicate results from (3) and (4) using rank fusion techniques”
↩︎ Checkpoint - 3.
“Chunk documents by heading - containing only the content from each subheading”
↩︎ Chunking: size, boundaries and the context problem“Some embedding models have fixed input token limits.”
↩︎ Chunking: size, boundaries and the context problem