What you will be able to do
- Explain how an agent decides at runtime whether and how to retrieve, and what guardrails that requires
- Expose an AI Search index to an LLM as a retriever tool
- Choose between Genie, the Databricks SQL MCP server and Unity Catalog functions for dynamic structured queries
- Select pre-inference or just-in-time retrieval for a given use case
1.Retrieval as a decision the model makes
Pre-inference retrieval means code fetches context on every request before the LLM runs. Just-in-time agentic retrieval reverses that. Retrieval becomes a tool, and the model decides whether to call it. Databricks calls this a single-agent system: one LLM that 'adaptively decides which tools to use, when to make more LLM calls, and when to stop.'
The same agent behaves differently when the user asks about their order. It calls a function like lookup_order(customer_id, order_id). If the tool returns 'invalid order number,' it retries or asks the user for the correct ID. The AI Search retrieval quality guide lists the abilities this gives you: the agent 'reasons about whether retrieval is needed,' can reformulate queries based on the first results, can combine retrieval with other tools, and can 'Retry failed retrievals with modified queries.' A fixed chain can do none of these.
# Agent decides when to search and what filters to apply
# Based on conversation context and user intent
agent = create_agent(
tools=[vector_search_tool, calculator, web_search],
instructions="Retrieve relevant docs only when needed, apply appropriate filters"
)Checkpoint 1 of 6· Check yourself
Which design concern applies to just-in-time agentic retrieval but not to a deterministic RAG chain?
When the model controls tool calls, it can call the same tool over and over. Databricks calls for iteration limits or timeouts. The other options apply to both strategies.
“Infinite loops can occur in any tool-calling scenario, so set iteration limits or timeouts.”Source: docs.databricks.com
2.Turning an AI Search index into a tool
For unstructured data, the agent's retrieval tool queries an AI Search index. Databricks offers two main routes. The quickest is the Databricks-managed AI Search MCP server at /api/2.0/mcp/ai-search/{catalog}/{schema}/{index_name}, which needs an index that uses Databricks-managed embeddings. The agent must be granted SELECT on the index's Unity Catalog securable. The second route is the AI Bridge packages (databricks-langchain, databricks-openai), which wrap an index as a VectorSearchRetrieverTool.
With a tool, the model sees only a name and a description, so the description is how it decides when to retrieve. Databricks advises you to 'Provide a descriptive tool_description to help the agent understand the tool and determine when to invoke it.'
from databricks_langchain import VectorSearchRetrieverTool, ChatDatabricks
# Initialize the retriever tool.
vs_tool = VectorSearchRetrieverTool(
index_name="catalog.schema.my_databricks_docs_index",
tool_name="databricks_docs_retriever",
tool_description="Retrieves information about Databricks products from official Databricks documentation."
)Checkpoint 2 of 6· Fill the gap
Which method hands the retriever to the LLM so the model can choose to call it?
# Bind the retriever tool to your Langchain LLM of choice
llm = ChatDatabricks(endpoint="databricks-claude-sonnet-4-5")
llm_with_tools = llm. ? ([vs_tool])bind_tools registers the retriever as a tool the model may call. It does not run a search. After binding, the LLM decides at inference time whether to call it.
Sources3
3.Dynamic queries against Delta and Unity Catalog tables
Just-in-time retrieval is not limited to vector search. When the answer is in rows of a table, such as last quarter's revenue or a specific customer's record, the agent can query governed tables live at the moment it needs the data. Databricks gives three mechanisms, and they differ in who writes the SQL.
Genie is the recommended default for querying Unity Catalog tables. A Genie Agent is a collection of up to 25 tables that Genie keeps in context and queries using natural language. Agents reach it through the managed MCP URL /api/2.0/mcp/genie/{genie_space_id}. One detail is easy to miss: the MCP route calls Genie as a tool, so conversation history is not passed along. If you need to pass existing context deterministically, call Genie as an agent in a multi-agent system instead.
The Databricks SQL MCP server (/api/2.0/mcp/sql) runs AI-generated SQL through a SQL warehouse, governed by Unity Catalog permissions. Queries run asynchronously: the agent starts a query, then polls until it completes. Databricks positions it for development and data engineering, meaning running SQL you or your coding agent already wrote. For natural-language business questions it points you to Genie instead.
Unity Catalog functions fit queries that are known ahead of time. You write the SQL, and the agent only supplies the parameters.
CREATE OR REPLACE FUNCTION main.default.lookup_customer_info(
customer_name_input STRING COMMENT 'Name of the customer whose info to look up'
)
RETURNS STRING
COMMENT 'Returns metadata about a particular customer, given the customer''s name, including the customer''s email and ID. The
customer ID can be used for other queries.'
RETURN SELECT CONCAT(
'Customer ID: ', customer_id, ', ',
'Customer Email: ', customer_email
)
FROM main.default.customer_data
WHERE customer_name = customer_name_input
LIMIT 1;| Mechanism | Who writes the query | Best fit | Connection |
|---|---|---|---|
| Genie Agent (managed MCP) | Genie, from natural language | Agent questions over Unity Catalog tables; the recommended default | https://<workspace-hostname>/api/2.0/mcp/genie/{genie_space_id} |
| Databricks SQL MCP server | The agent generates SQL directly | Running AI-generated SQL through a SQL warehouse, governed by Unity Catalog permissions | https://<workspace-hostname>/api/2.0/mcp/sql |
| Unity Catalog function | The developer, in advance | Queries known ahead of time where the agent supplies parameters | e.g. main.default.lookup_customer_info |
Checkpoint 3 of 6· Match them up
Match each just-in-time retrieval mechanism to the situation it fits.
Tap a term, then the definition that fits it.
The deciding factor is who writes the query and what kind of data is being retrieved: fixed SQL with parameters, natural-language analytics, a specific known query, or semantic search over documents.
“Create a structured retrieval tool using Unity Catalog SQL functions when the query is known ahead of time and the agent provides the parameters.”Source: docs.databricks.com
Checkpoint 4 of 6· Exam question
A context engineer is deciding how a sales analytics agent should answer ad hoc questions like "what were total bookings by region last week?" against a curated set of Unity Catalog tables. Which design reflects just-in-time agentic retrieval rather than pre-inference retrieval?
Correct answer: A — The agent is connected to a Genie space that generates and executes a SQL query against the tables when the question arrives
- A. Correct. Genie translates the question into a SQL query and executes it against the live Delta tables at the moment the question is asked, which is the defining trait of just-in-time agentic retrieval.
- B. Incorrect. Matching against pre-embedded schema summaries is a pre-inference, embedding-based lookup rather than a dynamic query executed against current data.
- C. Incorrect. Querying a batch-built AI Search index means the answer reflects whatever was pre-aggregated earlier, which is pre-inference retrieval, not a live query.
- D. Incorrect. Reading a cached summary written the prior evening returns pre-computed content rather than dynamically querying the underlying tables at request time.
4.Selecting a retrieval strategy for a use case
Choosing a strategy means reading the use case for three signals. First, does every request need the same kind of context? Second, can you write the retrieval path in advance? Third, is predictability worth more than adaptability? Databricks starts from the most conservative position: 'When building any AI-powered application, start simple.' A deterministic chain with up-front retrieval is the baseline. Move to agentic retrieval when requests vary enough that the model must choose its source, rewrite its query, or skip retrieval altogether.
Real systems often mix both. In Databricks' call-center example, a customer asks to return their last order. The agent plans, then 'queries the order database to retrieve the relevant order and references a policy document.' That is one live structured lookup and one unstructured reference, both chosen at runtime for this request.
| Use-case signal | Lean toward | Reason |
|---|---|---|
| Well-defined task, same retrieval for every request, audit required | Deterministic chain (pre-inference retrieval) | Highest predictability and auditability; fewer LLM calls for orchestration |
| Varied queries within one domain; only some need data | Single-agent system (just-in-time tool calls) | The agent chooses which, if any, tools to call |
| Tool result may fail and need a retry or a corrected input | Single-agent system (just-in-time tool calls) | The agent can loop through tool invocations to refine results |
| Large or cross-functional domains with distinct logic | Multi-agent system | Highly modular and scales to big domains, but harder to trace and debug |
Checkpoint 5 of 6· Check yourself
A support assistant gets a mix of general policy questions and requests about a specific customer's live order status, sometimes with a mistyped order ID. Which retrieval strategy fits?
Requests vary, only some need live structured data, and a bad order ID calls for a retry. A single-agent system decides per request which tool to call and can loop until it gets valid data.
“Certain queries or conditions can warrant tool usage such as deciding when to fetch customer data.”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
Which statement correctly defines pre-inference retrieval as distinct from just-in-time agentic retrieval?
Correct answer: A — Pre-inference retrieval loads or indexes content, such as embeddings, ahead of time so it is ready to search before the user query arrives
- A. Correct. Pre-inference retrieval refers to embedding or otherwise indexing content up front, so the search structure already exists and is simply queried when a request comes in.
- B. Incorrect. Issuing a live query at request time describes just-in-time agentic retrieval, the opposite of preparing data ahead of time.
- C. Incorrect. Requiring a tool call during the reasoning loop is the hallmark of just-in-time agentic retrieval, not pre-inference retrieval.
- D. Incorrect. Pre-inference retrieval commonly applies to unstructured document collections indexed as embeddings; it is not limited to structured tables.
Sources1
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Connecting an agent to Genie through the managed MCP server carries the conversation history into Genie.Why is that wrong?
The MCP route calls Genie as a tool, so history is not passed. To pass existing conversation context deterministically, use Genie as an agent in a multi-agent system.
Covered in Dynamic queries against Delta and Unity Catalog tables
2.For natural-language business questions over Delta tables, let the agent write SQL directly through the Databricks SQL MCP server.Why is that wrong?
Databricks reserves the SQL MCP server for running specific, already-written queries and points analytics questions in natural language to Genie.
Covered in Dynamic queries against Delta and Unity Catalog tables
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“The LLM adaptively decides which tools to use, when to make more LLM calls, and when to stop.”
↩︎ Retrieval as a decision the model makes“When building any AI-powered application, start simple.”
↩︎ Selecting a retrieval strategy for a use case“The agent queries the order database to retrieve the relevant order and references a policy document.”
↩︎ Selecting a retrieval strategy for a use case“If the user asks a simple question ("What is our returns policy?"), the agent might respond directly from the LLM's knowledge.”
↩︎ Prediction“Infinite loops can occur in any tool-calling scenario, so set iteration limits or timeouts.”
↩︎ Checkpoint“Certain queries or conditions can warrant tool usage such as deciding when to fetch customer data.”
↩︎ Checkpoint - 2.
“Agent reasons about whether retrieval is needed.”
↩︎ Retrieval as a decision the model makes“Retry failed retrievals with modified queries.”
↩︎ Retrieval as a decision the model makes - 3.
“Provide a descriptive tool_description to help the agent understand the tool and determine when to invoke it.”
↩︎ Turning an AI Search index into a tool“Grant the agent SELECT on the index's Unity Catalog securable.”
↩︎ Turning an AI Search index into a tool - 4.
“If your agent needs to query data in Unity Catalog tables, Databricks recommends using Genie Agents.”
↩︎ Dynamic queries against Delta and Unity Catalog tables“The managed MCP server for Genie invokes Genie as an MCP tool, which means history isn't passed when invoking Genie APIs.”
↩︎ Exam trap 1“Create a structured retrieval tool using Unity Catalog SQL functions when the query is known ahead of time and the agent provides the parameters.”
↩︎ Checkpoint - 5.
“Queries run asynchronously: the agent calls the tool to start a query, then polls until the response completes.”
↩︎ Dynamic queries against Delta and Unity Catalog tables“For analytics use cases, where a user asks a business question in natural language, use the Genie One MCP server instead.”
↩︎ Exam trap 2