What you will be able to do
- Choose between full-context prompting, indexed RAG, and agentic search for a given corpus size and change rate
- Repair chunking failures where the retrieved passage lacks the heading, entity, or defined term it depends on
- Combine BM25 with embeddings, then rerank, to fix exact-identifier misses without flooding the prompt
- Use citations, search result blocks, and uncertainty instructions to produce answers that can be checked
- Find which pipeline stage caused a bad answer and apply the fix at that stage
Key concept
Fix the stage that lost the information — A grounded answer can only use text that survives every stage: chunking, indexing, retrieval, selection, then generation. When a fact is lost at an early stage, no later stage can bring it back. Diagnose which stage dropped it and change that stage.
1.Full context, indexed RAG, or agentic search
Every knowledge-grounded application first has to decide how source text reaches the model. There are three options, and the choice depends on how big the corpus is and how fast it changes. The smallest case needs no retrieval at all. If a knowledge base is under about 200,000 tokens, roughly 500 pages, you can put the whole thing in the prompt. Prompt caching then makes repeated requests cheaper and faster. Below that size, a retrieval pipeline only adds ways to miss something.
Indexed RAG is the usual answer once the corpus is too large for the context window. The work happens ahead of time. You split the corpus into chunks of a few hundred tokens, embed them into vectors, and store them in a vector database. At query time the system finds the chunks most similar to the question and adds them to the prompt. This scales to very large corpora because the expensive steps run once, before any question arrives.
That same advantage is also the limitation: an index is a snapshot from its last build. Take a monorepo that changes hundreds of times a day. With a nightly rebuild, a function renamed this morning is still listed under its old name. Rebuilding every 30 minutes only shrinks the window. Ranking recent chunks higher still ranks within the same old snapshot. Re-chunking changes how the text is cut, not how old it is. The direct fix is agentic search: give the model tools such as glob, grep, and file-read over the live files, so retrieval happens when the question is asked.
Agentic search has its own costs. Runtime exploration is slower than reading a precomputed index. An agent with poorly designed tools or vague guidance wastes context on dead ends or misses the key file. The step-by-step exploration does have a benefit: each result tells the agent where to look next. A practical middle ground is a hybrid. Load stable, known-relevant material up front and let the agent explore the rest. Bounding how long that exploration runs is a latency question, covered in latency engineering.
| Architecture | Wins when | Main cost |
|---|---|---|
| Whole corpus in the prompt (with prompt caching) | Knowledge base under about 200,000 tokens | Does not scale past the context window |
| Indexed RAG (chunks, embeddings, vector database) | Large corpus that changes slowly compared with the rebuild schedule | Answers can be stale; context is lost at chunk boundaries |
| Agentic search (tools that fetch content at runtime) | Content changes constantly, or the answer takes several dependent lookups | Slower, and needs good tools and guidance |
Checkpoint 1 of 9· Match them up
Match each retrieval architecture to the knowledge-grounded application it fits best.
Tap a term, then the definition that fits it.
Below about 200,000 tokens, the whole corpus fits in the prompt, and caching makes repeated requests cheap. A large corpus that changes slowly suits a prebuilt index. Content that changes constantly needs tools that read the live files when the question is asked, because any index is a snapshot from its last build.
“Returning search results from your custom tools lets you build dynamic RAG applications: tools fetch content at runtime”Source: platform.claude.com
Checkpoint 2 of 9· Exam question
Russet Learning runs an internal engineering assistant that answers developer questions about its monorepo, which receives hundreds of commits a day. The assistant retrieves from a vector index that is rebuilt nightly. Developers report that answers cite functions that were renamed earlier the same morning, and the logs show the retrieved chunks contain only the pre-rename names. Engineers have noted that a full re-embedding of the repository is slow and costly, and a few asked questions within minutes of merging their own changes and still got stale answers. The team accepts a few extra seconds of latency per question if answers reflect the code as it currently stands. Which change most directly fixes the staleness?
Correct answer: D — Give the agent glob, grep and file-read tools over the live checkout, keeping retrieval tied to the files as they stand when each question is asked.
- A. A shorter schedule narrows the gap but does not close it. With hundreds of commits a day, a rename stays unsearchable until the next build finishes, so answers between rebuilds still cite old names, and re-embedding a monorepo this often adds cost.
- B. Timestamp ranking only helps when the index holds both the old and the new version of a function. A rename made after the nightly build is in no chunk, so there is nothing newer to rank higher and the old name is still cited.
- C. Smaller chunks with more overlap only duplicate text already in the nightly index. A rename made after the last build is in no chunk, so the extra overlap spreads the old name across more chunks instead.
- D. Correct. Search tools over the live checkout read the files as they currently are, so a rename made that morning is found immediately and there is no index to go stale. The tradeoff is extra tool-call turns per question.
2.Chunking that keeps the context a passage needs
Chunking is where context most often disappears. Consider a chunk from an SEC filing that says "The company's revenue grew by 3% over the previous quarter." On its own, it does not say which company or which quarter. A question about ACME's Q2 revenue may never retrieve it. If it is retrieved, the model still cannot tell what it refers to. Contracts fail the same way. A fixed 200-token window can catch the dollar figure of an indemnification cap but leave the clause heading and the defined term in neighbouring chunks.
Fixed-size chunks will not solve this by getting smaller. A 100-token chunk with no overlap focuses each embedding on a single statement, but it removes even more of the surrounding text the statement depends on. Three settings decide what each chunk contains: chunk size, where the boundaries fall, and how much neighbouring chunks overlap. For structured documents, cut along the document's own structure. Split contracts at section and clause boundaries, with a modest overlap, so each passage keeps its title and the whole clause.
A second fix separates the unit you search from the unit you send to the model. Index small chunks so matching stays precise. Then, before generation, replace each hit with its enclosing section, so the model sees the heading and definitions with the figure. Neither a larger embedding model nor a reranker can do this. Both work only on the text inside the chunk, and they cannot add a heading that was cut away. A generation-time instruction to "cite the clause title" has the same problem: the title is not in the model's context.
Contextual Retrieval puts the missing context back into each chunk before indexing. Claude reads the whole document and writes a 50–100 token description that places one chunk within it. That text is prepended to the chunk before embedding and before building the BM25 index. The SEC chunk becomes "This chunk is from an SEC filing on ACME corp's performance in Q2 2023…". With prompt caching, the document is cached once and reused for every chunk, so this preprocessing is cheap. Contextual embeddings alone cut top-20 retrieval failures by 35%. The method helped with every embedding model tested, though some models gained more than others.
Chunk structure matters at citation time too. With custom content documents and search result blocks, the text blocks you provide are the units Claude can cite. Splitting long content into logical blocks gives Claude finer citation boundaries.
Checkpoint 3 of 9· Check yourself
A contract-review RAG system splits documents into fixed 200-token chunks. Retrieved chunks often hold an indemnification cap's dollar figure, but the clause heading and defined term are in neighbouring chunks. Answers then assign the cap to the wrong clause. Which change fixes the stage that lost the context?
The context was lost at chunking, so the fix belongs at chunking. Cutting along the contract's own structure keeps the title and the whole clause together. Smaller chunks strip away even more context. A reranker and a generation-time instruction can only work with the text already in the chunk, and the heading is not there.
“Chunk boundaries: Consider how you split your documents into chunks. The choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance”Source: www.anthropic.com
Checkpoint 4 of 9· Exam question
Quarry Grocers runs a legal-research tool for its procurement team that answers questions about supplier contracts. The system splits every contract into fixed 200-token chunks. Answers about indemnification caps are often wrong: the retrieved chunk contains the dollar figure, but the clause heading and the defined term it refers to sit in neighboring chunks. A review of 50 failed answers shows the correct chunk ranked first in nearly every case. The team already runs generation at temperature 0, and its prompt asks the model to name the clause title and defined terms, yet the model replies that they are not stated. Contracts follow a consistent layout of numbered sections with headings. Select TWO changes that best address this.(Select 2)
Correct answers: A, D — Index small chunks for matching, then replace each retrieved hit with its enclosing section before the text reaches the model.; Split contracts along section and clause boundaries with a modest overlap, so each passage carries its title and the full clause text.
- A. Correct: matching on small chunks keeps retrieval precise, and swapping each hit for its enclosing section hands the model the clause title and the definitions that sit outside the 200-token window.
- B. The retrieved chunk already contains the figure, so retrieval is not the weak point. A larger embedding model or a reranker only reorders or rescores chunks that still lack the title and definition, which is text the model never receives.
- C. Temperature affects sampling randomness, and an instruction to cite the clause title and defined terms cannot work when those words are absent from the retrieved context. The model has nothing to cite, so the wrong answers persist.
- D. Correct: chunking along section and clause boundaries keeps each clause whole with its title, so a retrieved passage is understandable without the neighboring chunks that fixed 200-token splits cut away.
- E. Shorter chunks cut contracts into even smaller fragments, and with no overlap the title and definition end up in separate chunks more often. Retrieval may look tighter, but the missing-context problem gets worse.
3.Hybrid search and reranking
Dense embeddings capture meaning, but they can miss exact strings. Ask a support index about "Error code TS-999". An embedding model may return content about error codes in general and skip the one document that contains TS-999. Error codes, version strings, part numbers, and other identifiers are exactly the tokens that semantic similarity tends to blur.
BM25 is the lexical counterpart. It is a ranking function based on TF-IDF that scores exact word and phrase matches. It adjusts for document length and caps how much repeated terms count, so common words do not dominate. A hybrid pipeline runs both retrievers over the same chunks. BM25 takes the top chunks by exact match, embeddings take the top chunks by meaning, and rank fusion merges the two lists and removes duplicates. Contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49%.
Several tempting fixes stay on the dense-only path and fail. Lowering the similarity threshold admits more borderline chunks, but the exact-match document still does not rank. Raising top-k to 200 and letting the model pick hands it a pile of mostly irrelevant text. More chunks raise the chance the right one is included, but extra information distracts the model, so there is a limit. A system-prompt rule to "prefer chunks containing the exact code" acts at generation time. It cannot help when retrieval never returned the chunk.
Reranking resolves the conflict between wide recall and a short prompt. First, retrieve broadly. Anthropic's experiments took the top 150 candidates. A reranking model then scores each candidate against the query, and only the best few go to the model; they kept the top 20. Because the model processes less text, answers get better and cost and latency go down. Adding reranking to Contextual Retrieval raised the reduction in failed retrievals from 49% to 67%. Tool-based retrieval follows the same rule: return only the most relevant results so the context does not overflow.
| Component | Catches | Misses on its own |
|---|---|---|
| Dense embeddings | Paraphrases and matches by meaning | Exact identifiers and technical terms |
| BM25 | Exact tokens such as error codes and version strings | Paraphrased questions with no shared words |
| Rank fusion | Combines both lists without duplicates | Does not shorten what reaches the prompt |
| Reranker | Narrows a wide candidate set (top 150) to a short context (top 20) | Cannot recover a chunk that neither retriever returned |
Checkpoint 5 of 9· Check yourself
Users of a support assistant search for "Error code TS-999". The retriever uses dense embeddings only. It returns general articles about error codes, and the one document that mentions TS-999 never appears. Which change fixes this miss?
Embeddings can blur exact identifiers, so the chunk is lost at the retrieval stage. BM25 scores exact string matches, so it can bring that chunk back. A system-prompt rule applies only at generation, after retrieval has already dropped the chunk. A lower threshold or a larger top-k still relies on dense similarity, which doesn't rank the exact match.
“It's particularly effective for queries that include unique identifiers or technical terms.”Source: www.anthropic.com
Checkpoint 6 of 9· Check yourself
After adding hybrid search, the relevant chunk usually shows up in the fused top 150, often somewhere between rank 50 and rank 150. Sending all 150 chunks makes answers worse, slower, and more expensive. What should the pipeline do?
Reranking keeps the wide recall of 150 candidates and still sends only a short context to the model. Cutting fusion to its top 20 would drop the relevant chunk whenever it ranks lower than that. Sending everything distracts the model. Dropping BM25 would lose exact-match recall again.
“This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%.”Source: www.anthropic.com
Checkpoint 7 of 9· Exam question
Saffron Energy runs a product-support bot for its smart-meter and inverter customers, retrieving manuals and release notes using only dense embeddings. For queries like "error E-4021 on firmware 3.2.1", the top results discuss generic firmware errors. A literal search of the corpus for "E-4021" confirms the correct release-note page exists, yet it ranks far below position 200 and never reaches the model. In testing, a looser similarity cutoff only admitted more generic firmware pages. Token spend per answer is already near budget, and the system prompt already tells the model to cite exact codes. Other phrasings retrieve well, and the team wants to keep that behaviour. Select TWO changes that address this retrieval failure.(Select 2)
Correct answers: A, D — Combine the dense and BM25 result lists using reciprocal rank fusion, then rerank the merged candidates for the prompt.; Add a BM25 keyword index next to the dense retriever, so literal tokens like the error code and version string match.
- A. Correct: once a lexical ranking exists beside the dense one, rank-based fusion merges the two lists without needing comparable raw scores, and the reranker then sees candidates from both.
- B. A larger dense top-k still ranks by semantic similarity, so the chunk with the exact code may not be retrieved at all. It also adds 200 chunks of cost and distraction for the model.
- C. Loosening the threshold only admits more semantically similar chunks. Nothing in dense scoring rewards a literal match on the code or version, so the exact-code document is not reliably surfaced.
- D. Correct: keyword matching catches rare identifiers such as an error code or version string, which embeddings blur into semantically similar generic firmware errors.
- E. A prompt rule can only steer the model among the chunks retrieval already returned. If the exact-code document is not among them, the instruction has nothing to act on.
4.Grounding answers: citations, uncertainty, and confidence signals
Retrieval decides what the model can see. Grounding decides whether the answer can be checked against it. Citations make it checkable. When citations are enabled on documents, the response is split into text blocks. Each block can carry citations that point to the exact passage supporting it: character ranges for plain text, page ranges for PDFs, and block indices for custom content. Scanned PDFs without extractable text cannot be cited. The cited_text field does not count toward output tokens.
For RAG output, wrap each retrieved passage in a search result content block rather than pasting it as plain text. Each citation then carries the source and title you set, so an answer traces back to a URL or an internal ID such as kb://article-1234. Search results can come from a custom tool's result or sit directly in the user message. Citations are disabled by default, and every search result in a request must use the same setting.
{
"type": "search_result",
"source": "https://example.com/article", // Required: Source URL or identifier
"title": "Article Title", // Required: Title of the result
"content": [
// Required: Array of text blocks
{
"type": "text",
"text": "The actual content of the search result..."
}
],
"citations": {
// Optional: Citation configuration
"enabled": true // Enable/disable citations for this result
}
}Prompt format can quietly turn citations off. A prompt that asks for only a JSON object still succeeds, but the response may contain few or no citations. To get a citation for each extracted value, ask for one field: value line per field and parse the lines into structured data in your own code.
Saying "I don't know" has to be designed in. Give Claude explicit permission to admit uncertainty. For example: "If the report lacks necessary information, say 'I don't have enough information to confidently assess this.'" Tell it to use only the provided documents, not its general knowledge. For long documents, ask it to extract word-for-word quotes first, and to state "No relevant quotes found." when there are none. Handle a search that returns nothing the same way. Return a plain text block such as No results found. instead of raising an error, so Claude can explain the empty result and the conversation continues.
Confidence signals come from checks you can see, not from the model's tone. Three are available:
- Whether each claim has a citation at all.
- Whether a claim survives a verification pass, where Claude looks for a supporting quote for each claim and removes any claim it cannot support.
- Whether answers agree across best-of-N runs, since inconsistency across outputs can indicate hallucination.
A claim with no citation, removed during verification, or that changes between runs is a low-confidence claim. Show it that way to users.
Checkpoint 8 of 9· Fill the gap
You wrap each retrieved passage in a search result block so that every citation traces back to a URL or an internal ID such as kb://article-1234. Which field name completes the block?
{
"type": "search_result",
" ? ": "https://example.com/article", // Required: Source URL or identifier
"title": "Article Title", // Required: Title of the result
"content": [
// Required: Array of text blocks
{
"type": "text",
"text": "The actual content of the search result..."
}
],
"citations": {
// Optional: Citation configuration
"enabled": true // Enable/disable citations for this result
}
}The required source field takes any stable string, such as a URL or an internal ID. Claude returns it, along with title, on every citation that draws on the result, so the answer can be traced to where it came from. The block has no url, document_id, or context field.
5.Diagnosing retrieval-quality failures by stage
A wrong grounded answer is a symptom. The cause sits in one stage of the pipeline, and each stage leaves its own evidence. Start by checking whether the right passage reached the model's context. If it never did, the fault is upstream: chunking or retrieval. If it did and the answer is still wrong or unsupported, look at selection and generation. A team that tunes prompts for a retrieval miss, or swaps embedding models for a generation error, makes changes that cannot work.
| Stage | Symptom | Fix at this stage |
|---|---|---|
| Chunking / indexing | Right chunk retrieved, but it lacks the entity, heading, or defined term | Structure-aware boundaries with overlap; contextual chunks; pass the enclosing section |
| Index freshness | Answers cite names or facts that changed after the last build | Agentic search with tools that read live content at runtime |
| Retrieval | Relevant chunk not in the top 20 (exact identifiers missed) | Add BM25 alongside embeddings and fuse the rankings |
| Selection / reranking | Right chunk retrieved but crowded out, or the prompt is bloated | Rerank a wide candidate set down to a short top-K |
| Generation / grounding | Passage was in context, but the claim is uncited or unsupported | Enable citations, check every claim has one, verify claims against quotes |
Measure retrieval on its own, apart from answer quality. Anthropic's experiments used 1 minus recall@20: the share of relevant documents that fail to appear in the top 20 retrieved chunks. That number tells you whether a change to chunking, embeddings, BM25, or the reranker helped retrieval, without generation getting in the way. Run it on a labelled question set before and after every pipeline change. Building that harness belongs to evaluation system design.
Check grounding on its own as well. The API does not guarantee that every extracted field has a value or that every value has a citation, so check both in code. A citation that exists can still fail to support its sentence. Compare cited_text against the claim it is attached to, or have Claude find a supporting quote for each claim and remove claims it cannot support.
Checkpoint 9 of 9· Check yourself
Request logs for a wrong answer show that the chunk holding the correct quarterly revenue figure was retrieved and placed in the prompt. Even so, the answer states a different figure that appears nowhere in that chunk. Which change acts on the stage that failed?
The right passage reached the model's context, so chunking, retrieval and selection did their job. The unsupported claim is a generation and grounding failure. Citations and claim-by-claim quote verification act on that stage. BM25, a different embedding model and reranking all change retrieval or selection, and those stages did not fail here.
“have Claude verify each claim by finding a supporting quote after it generates a response”Source: platform.claude.com
Sources
(5)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“tools fetch content at runtime, and Claude cites it in the response”
↩︎ Full context, indexed RAG, or agentic search“Break long content into logical text blocks to give Claude finer citation boundaries”
↩︎ Chunking that keeps the context a passage needs“Result limits: Return only the most relevant results to avoid context overflow”
↩︎ Hybrid search and reranking“each citation carries the source and title you provided”
↩︎ Grounding answers: citations, uncertainty, and confidence signals“when a search fails or returns nothing, return a plain text block describing the outcome”
↩︎ Grounding answers: citations, uncertainty, and confidence signals“tools fetch content at runtime, and Claude cites it in the response”
↩︎ Exam trap 4“Returning search results from your custom tools lets you build dynamic RAG applications: tools fetch content at runtime”
↩︎ Checkpoint - 2.https://www.anthropic.com/engineering/contextual-retrievalSecondary source
“you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods”
↩︎ Full context, indexed RAG, or agentic search“The choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance”
↩︎ Chunking that keeps the context a passage needs“The resulting contextual text, usually 50-100 tokens, is prepended to the chunk before embedding it and before creating the BM25 index.”
↩︎ Chunking that keeps the context a passage needs“Response generation may be improved by passing it the contextualized chunk and distinguishing between what is context and what is the chunk.”
↩︎ Chunking that keeps the context a passage needs“Whereas Contextual Retrieval improves performance across all embedding models we tested, some models may benefit more than others.”
↩︎ Chunking that keeps the context a passage needs“It's particularly effective for queries that include unique identifiers or technical terms.”
↩︎ Hybrid search and reranking“Combine and deduplicate results from (3) and (4) using rank fusion techniques;”
↩︎ Hybrid search and reranking“Reranking is a commonly used filtering technique to ensure that only the most relevant chunks are passed to the model.”
↩︎ Hybrid search and reranking“This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%.”
↩︎ Hybrid search and reranking“However, more information can be distracting for models so there's a limit to this.”
↩︎ Hybrid search and reranking“measures the percentage of relevant documents that fail to be retrieved within the top 20 chunks”
↩︎ Diagnosing retrieval-quality failures by stage“traditional RAG solutions remove context when encoding information, which often results in the system failing to retrieve the relevant information from the knowledge base.”
↩︎ Key concept“While embedding models excel at capturing semantic relationships, they can miss crucial exact matches.”
↩︎ Exam trap 1“it can lead to problems when individual chunks lack sufficient context.”
↩︎ Exam trap 2“adding generic document summaries to chunks (we experimented and saw very limited gains)”
↩︎ Prediction“Chunk boundaries: Consider how you split your documents into chunks. The choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance”
↩︎ Checkpoint - 3.
“runtime exploration is slower than retrieving pre-computed data”
↩︎ Full context, indexed RAG, or agentic search“Without proper guidance, an agent can waste context by misusing tools, chasing dead-ends, or failing to identify key information.”
↩︎ Full context, indexed RAG, or agentic search“the most effective agents might employ a hybrid strategy, retrieving some data up front for speed, and pursuing further autonomous exploration at its discretion”
↩︎ Full context, indexed RAG, or agentic search - 4.
“Citations return the exact passages that support each claim, so you can verify answers and surface sources to your users.”
↩︎ Grounding answers: citations, uncertainty, and confidence signals“PDFs that are scans of documents and do not contain extractable text are not citable.”
↩︎ Grounding answers: citations, uncertainty, and confidence signals“Check that each field has a value and at least one citation.”
↩︎ Diagnosing retrieval-quality failures by stage“A request with such a prompt succeeds, but the response might contain few or no citations.”
↩︎ Exam trap 3 - 5.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinationsOfficial docs
“Explicitly give Claude permission to admit uncertainty. This technique can drastically reduce false information.”
↩︎ Grounding answers: citations, uncertainty, and confidence signals“Inconsistencies across outputs could indicate hallucinations.”
↩︎ Grounding answers: citations, uncertainty, and confidence signals“You can also have Claude verify each claim by finding a supporting quote after it generates a response.”
↩︎ Diagnosing retrieval-quality failures by stage“have Claude verify each claim by finding a supporting quote after it generates a response”
↩︎ Checkpoint