Contextual Retrieval, Reranking and Just-in-Time Agentic Search

1.Fixing the isolated chunk: contextual embeddings and contextual BM25

Chunking makes large corpora searchable, but it removes context. The Contextual Retrieval post uses an SEC-filing example. A user asks for ACME Corp's revenue growth in Q2 2023. The relevant chunk reads "The company's revenue grew by 3% over the previous quarter." Nothing in that chunk names the company or the quarter, so neither embeddings nor BM25 can reliably connect it to the question.

Contextual Retrieval fixes this when the corpus is ingested. Claude writes a short note placing each chunk within its parent document, usually 50–100 tokens. That note is prepended to the chunk before it is embedded and before the BM25 index is built. This is the prompt Anthropic used:

<document>
{{WHOLE_DOCUMENT}}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{{CHUNK_CONTENT}}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.

Generic summaries didn't help much. Context written for each chunk did. Across codebases, fiction, ArXiv and science papers, contextual embeddings cut the top-20 retrieval failure rate by 35%. Adding contextual BM25 brought the reduction to 49%. The cost is paid once, at ingestion. With prompt caching, the full document is cached once and reused for each of its chunks, which puts the post's estimate at $1.02 per million document tokens.

The post lists a few things to tune for your data. Chunk size, boundaries and overlap all affect results. Custom contextualizer prompts can do better than the generic one, for example by including a glossary of terms that are defined in other documents. Some embedding models also gain more than others. If you'd rather not build this pipeline yourself, Anthropic's embeddings guide lists Voyage contextualized chunk models (voyage-context-4, voyage-context-3) that fold document context into chunk vectors directly.

Sources12

2.Reranking, and choosing a stack by accuracy, latency and cost

Reranking adds a second stage at query time. Initial retrieval methods are tuned to be fast across millions of documents. A reranking model is slower but more precise, so it is run only on a small shortlist. Anthropic's experiment took the top 150 candidates and scored each one against the query with a reranker. It kept the top 20. Reranked contextual embeddings plus contextual BM25 cut the top-20 failure rate by 67%.

Each layer improves accuracy, and each one has a different cost. The cookbook measured them on 9 codebases and 248 queries:

The cookbook prices each layer. Contextual embeddings cost about $3 once, at ingestion, for this dataset. Hybrid search needs Elasticsearch infrastructure to run and maintain. Reranking adds 100–200ms of latency and about $0.002 in API cost to every query. From that, the cookbook gives three profiles:

Within reranking there is a further setting: how many chunks to rerank. Reranking more chunks improves accuracy and costs more latency. Anthropic's embeddings guide shows the same choice at the model level. Voyage offers rerank-2.5 for highest accuracy and rerank-2.5-lite, which is optimized for latency and cost.

Sources123

3.Beyond a pre-built index: just-in-time retrieval through tools

Everything so far builds an index ahead of time. Anthropic's context-engineering guidance describes another approach, which suits large databases and data that changes often. The agent holds lightweight references, such as file paths, stored queries and links, and loads data through tools only when it needs it. Claude Code works this way when analysing large databases. It writes targeted queries, stores the results and inspects them with head and tail, so the full data never has to enter the context window.

This retrieval is iterative. In the guide's words, each interaction yields context that informs the next decision, so the agent builds understanding step by step. The sources don't use the term "multi-hop retrieval" or compare it with single-hop lookup. What they do describe is an agent that keeps querying until it has enough. The cost is speed: runtime exploration is slower than retrieving pre-computed data, and without good tools and guidance an agent can waste context on dead ends.

A middle path is the hybrid strategy. Load some data up front for speed and let the agent explore the rest. Claude Code puts CLAUDE.md files into context at the start and uses glob and grep to fetch files as needed, which avoids stale indexes. The guide suggests this hybrid fits less dynamic content, such as legal or finance work. On the API side, a custom retrieval tool can return results as search_result blocks, which lets Claude cite your sources:

{
  "type": "search_result",
  "source": "https://example.com/article", // Required: Source URL or identifier
  "title": "Article Title", // Required: Title of the result
  "content": [
    // Required: Array of text blocks
    {
      "type": "text",
      "text": "The actual content of the search result..."
    }
  ],
  "citations": {
    // Optional: Citation configuration
    "enabled": true // Enable/disable citations for this result
  }
}

Sources45

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Prepending a generic summary of the whole document to each chunk gets the same benefit as Contextual Retrieval.Why is that wrong?

   Anthropic tested generic summaries and saw very limited gains. The improvement comes from short context that Claude writes for each individual chunk.

   Covered in Fixing the isolated chunk: contextual embeddings and contextual BM25

2. 2.Because reranking scores best, it should be the default for every production RAG system.Why is that wrong?

   Reranking adds latency and API cost to every query. For high-volume, cost-sensitive systems the cookbook recommends contextual embeddings alone.

   Covered in Reranking, and choosing a stack by accuracy, latency and cost