What you will be able to do
- Recognise an accurate but contextually mismatched response as a retrieval quality problem, not a generation problem
- Put the inference-time stages of a RAG chain in order and name the stage whose output feeds retrieval
- Identify query understanding (user query preprocessing) as the stage where user intent is resolved before context is retrieved
Key concept
Query understanding (user query preprocessing) — The first stage of a RAG chain at inference time. It turns the raw user query into a retrieval query that captures what the user actually means, so retrieval can fetch context that fits the request. It runs before retrieval, which makes it the only stage that can shape what gets retrieved.
1.Diagnosing an accurate but mismatched answer
Here is the scenario. An agent's answer is factually correct and faithful to the documents it was given, but it doesn't answer what the user meant. A user asks about returning their most recent order and gets a correct summary of the general returns policy. A user in a multi-turn chat asks a follow-up and gets an accurate answer to a different question. Nothing in the output is a hallucination. The facts are right. The context is wrong.
Databricks separates RAG quality issues into two types. Retrieval quality asks: "Are you retrieving the most relevant information for a given retrieval query?" Generation quality asks whether the LLM, given what it retrieved, produces an accurate, coherent and helpful response. Generation problems look like hallucinations or inconsistent output. A response that stays faithful to its context but misses the user's need points to the context itself, meaning what got retrieved and why.
Many people reach for the data pipeline first: chunking, parsing, embeddings. Databricks warns against assuming a clean split where retrieval issues mean updating the data pipeline and generation issues mean updating the chain. Retrieval quality depends on the data pipeline *and* on the RAG chain, and the chain's contribution includes user query transformation. The effect also carries downstream: generation quality "will invariably be impacted by poor retrieval". So when you see a mismatched answer at generation, the cause usually sits earlier in the chain.
Databricks' writing on agent harnesses describes the same failure from the operator's side: "When the harness pulls in the wrong information from memory or search systems, the model may confidently generate incorrect answers." The model is answering confidently from whatever it was handed. To fix the answer, you have to fix what it was handed.
Checkpoint 1 of 4· Check yourself
Which symptom points to a generation quality issue rather than a retrieval quality issue?
Hallucination and inconsistent output are how Databricks describes generation quality issues. The other three describe the wrong context reaching the model, which is a retrieval quality problem.
“Issues here can manifest as hallucinations, inconsistent output, or failure to directly address the user query.”Source: docs.databricks.com
2.The stages of the RAG chain at inference time
To find where intent should be resolved, lay out the chain in order. Databricks calls the sequence of steps that runs when a user submits a request the RAG chain:
1. (Optional) User query preprocessing. The query is formatted in a template, rewritten by another model, or mined for keywords. "The output of this step is a retrieval query which will be used in the subsequent retrieval step." 2. Retrieval. The retrieval query is embedded with the same model used for the document chunks, and the most similar chunks come back ranked. 3. Prompt augmentation. The user's query is combined with the retrieved context, "in a template that instructs the model how to use each component". 4. LLM generation. The model produces a response grounded in that augmented prompt. 5. (Optional) Post-processing. Business logic, citations or refinement are applied to the generated text.
Guardrails can run throughout, for example checking user permissions before data sources are accessed.
| Stage | Input | Output | Can it change which context is retrieved? |
|---|---|---|---|
| User query preprocessing | Raw user query | Retrieval query | Yes, it defines the query retrieval runs on |
| Retrieval | Retrieval query | Top-ranked chunks | It runs the search, but only on the query it receives |
| Prompt augmentation | User query plus retrieved chunks | Augmented prompt | No, retrieval has already happened |
| LLM generation | Augmented prompt | Response | No |
| Post-processing | Response | Refined response | No |
The table shows the constraint. Retrieval only ever sees the retrieval query. Anything the user meant that isn't in that query (the order they're referring to, the earlier turn a follow-up depends on, a time period) can't affect which chunks come back. Prompt augmentation does see the original user query, but by then the chunks are fixed. Telling the model there to "focus on what the user really meant" just asks it to work with context that may not contain the answer.
Checkpoint 2 of 4· Put it in order
Put the RAG chain inference stages in the order they run
- 1.LLM generation produces a grounded response
- 2.User query preprocessing produces a retrieval query
- 3.Post-processing applies business logic or citations
- 4.Prompt augmentation combines the user query with the retrieved context
- 5.Retrieval embeds the retrieval query and returns the most similar chunks
Preprocessing comes first because its output is the input to retrieval. Augmentation needs the retrieved context, generation needs the augmented prompt, and post-processing refines the generated text.
“The output of this step is a retrieval query which will be used in the subsequent retrieval step.”Source: docs.databricks.com
Sources3
3.Query understanding: the stage that resolves intent
Databricks' guide to improving RAG chain quality gives the first stage a name with a clear job: query understanding, which means analysing and transforming user queries to better represent intent and to pull out filters or keywords for retrieval. The guide says plainly that passing the raw query straight to retrieval is a shortcut. It "can work for some queries", but "it is generally beneficial to reformulate the query before the retrieval step." Query understanding sits at the beginning of the chain because everything after it depends on the context it lets retrieval find.
Agents follow the same pattern. In Databricks' call-center example, the customer asks to return their last order. The next step reads: "Given the intent of the query, the agent" plans to look up the user's recent order and check the return policy. Only after that does it query the order database and policy document. Intent is resolved into a plan first, and data gathering follows from that plan. If you skip that step, the agent might retrieve the return policy and nothing about the order. The answer would be accurate and still mismatched.
The system prompt only takes effect at prompt augmentation and generation, after retrieval has already run on the bare follow-up text. The conversation history has to be folded into the retrieval query during query understanding, for example by paraphrasing it, so that retrieval fetches context relevant to the whole conversation.
Checkpoint 3 of 4· Exam question
A support agent is asked 'What is the return window on my order?' and correctly retrieves and quotes the return policy for electronics, citing the exact policy document. The customer's order, however, is for a furniture item that has a different return window, so the answer is wrong for this customer even though every retrieved fact is accurate. Where in the agent pipeline should this be fixed?
Correct answer: B — Determine which product category the customer's order belongs to during the reasoning/planning step, before any retrieval call is issued
- A. Wrong. Reordering by date does not address the root cause: the agent retrieved documents for the wrong product category, not documents that were merely out of order.
- B. Correct. The order category is part of user intent. Resolving it during the initial reasoning/planning stage, before retrieval is invoked, ensures the retrieval call itself targets the right policy document.
- C. Wrong. Larger chunks would add more surrounding text to the electronics policy document, but the agent would still be retrieving the wrong document entirely.
- D. Wrong. Filtering the output after generation is reactive and does not prevent the wrong document from being retrieved and used to construct the answer in the first place.
- E. Wrong. Adding more historical versions to the index increases the chance of retrieving an even less relevant document; it does not resolve which category the customer meant.
Checkpoint 4 of 4· Check yourself
An agent pipeline returns accurate, well-grounded answers that often don't match what users meant. At which stage should user intent be resolved?
Query understanding is the stage at the beginning of the chain that turns the user query into a retrieval query. It is the last chance to influence which context is retrieved. Augmentation and post-processing both run after retrieval.
“it is generally beneficial to reformulate the query before the retrieval step.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Contextually mismatched answers are a data pipeline problem, so the fix is re-chunking or changing the embedding model.Why is that wrong?
The RAG chain affects retrieval quality too, through user query transformation. When the documents are sound and answers are accurate, the mismatch usually comes from the query that was sent to retrieval.
Covered in Diagnosing an accurate but mismatched answer
2.Intent can be resolved at prompt augmentation by adding instructions about what the user means to the prompt template.Why is that wrong?
Prompt augmentation combines the query with context that has already been retrieved, so it can't change which chunks were fetched. Intent has to be resolved in the preprocessing step whose output becomes the retrieval query.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Are you retrieving the most relevant information for a given retrieval query?”
↩︎ Diagnosing an accurate but mismatched answer“generation quality will invariably be impacted by poor retrieval”
↩︎ Diagnosing an accurate but mismatched answer“the RAG chain (for example, user query transformation, number of chunks retrieved, re-ranking)”
↩︎ Exam trap 1“the RAG chain (for example, user query transformation, number of chunks retrieved, re-ranking)”
↩︎ Prediction“Issues here can manifest as hallucinations, inconsistent output, or failure to directly address the user query.”
↩︎ Checkpoint - 2.https://www.databricks.com/blog/ai-harnessSecondary source
“When the harness pulls in the wrong information from memory or search systems, the model may confidently generate incorrect answers.”
↩︎ Diagnosing an accurate but mismatched answer - 3.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/fundamentals-inference-chain-ragOfficial docs
“The output of this step is a retrieval query which will be used in the subsequent retrieval step.”
↩︎ The stages of the RAG chain at inference time“with the retrieved context, in a template that instructs the model how to use each component”
↩︎ The stages of the RAG chain at inference time“The output of this step is a retrieval query which will be used in the subsequent retrieval step.”
↩︎ Exam trap 2 - 4.
“Using the user query directly as a retrieval query can work for some queries.”
↩︎ Query understanding: the stage that resolves intent“Analyzing and transforming user queries to better represent intent and extract relevant information, such as filters or keywords, to improve the retrieval process.”
↩︎ Key concept“it is generally beneficial to reformulate the query before the retrieval step.”
↩︎ Checkpoint - 5.
“Given the intent of the query, the agent”
↩︎ Query understanding: the stage that resolves intent