What you will be able to do
- Explain why a few-shot example has to earn its tokens by changing agent behaviour on cases the agent would otherwise get wrong
- Rank candidate examples by marginal contribution: covering untested tool paths, showing output structure, and handling ambiguous inputs
- Spot low-value candidates: near-duplicates, examples of paths that already work, and format examples that a schema already enforces
- Check an inclusion decision by comparing prompt versions on the same evaluation data, without leaking evaluation items into the prompt
Key concept
Marginal contribution of a few-shot example — The improvement an example adds to the agent's behaviour on top of what the instructions, tool descriptions and other examples already achieve, measured against the tokens it costs on every request. An example that only repeats behaviour the agent already gets right costs tokens and adds nothing.
1.Every example is paid for on every request
Exam questions on this objective follow the same pattern. You get an agent (a tool-calling agent, a RAG chain, or a Genie-style text-to-SQL agent), a list of candidate few-shot examples, and a token budget that cannot hold all of them. You have to say which examples go in and why. The wrong way to read it is "which examples are good?" Nearly every candidate is a correct, well-written example. The right question is: what does this example teach the model that nothing else in the prompt already teaches it?
Few-shot examples sit in the prompt, so every one of them is sent with every request. Databricks treats token consumption as a first-class measure of quality, next to correctness. The evaluation metrics include total, input and output token counts alongside latency. An example therefore has a fixed cost on each request, and its benefit only shows up on the requests whose behaviour it changes.
This gives you a working rule. Rank candidates by how many failing or unstable behaviours they fix, per token. Three kinds of candidate tend to rank highest, and the exam guide names all three: examples that cover a tool path the agent does not yet handle, examples that pin down an output structure, and examples that show what to do with an ambiguous input. Examples that rank lowest are near-duplicates of another selected example and examples of paths the agent already handles. Each of the following sections covers one of the high-value kinds.
Checkpoint 1 of 8· Check yourself
Why is "this example is correct and well written" not enough reason to include it in a budget-limited prompt?
Token counts are tracked as a cost metric next to quality metrics. An example that does not change behaviour only adds cost. The other options are invented limits or behaviours.
“What's the total count of tokens for LLM generations?”Source: docs.databricks.com
Checkpoint 2 of 8· Exam question
A context engineer is curating few-shot examples for a Genie space under a strict token budget. The candidate pool already has three examples that each demonstrate a straightforward single-table filter query returning the same output shape. A fourth candidate demonstrates a multi-step aggregation with a re-used calculation the finance team asks about weekly. Which action best reflects marginal-contribution reasoning under the budget?
Correct answer: A — Include the fourth example and drop one of the redundant single-table filter examples
Sources1
2.Cover the tool paths the agent does not yet handle
In a single-agent system the LLM chooses which tools to call, in what order, and when to stop. Databricks notes the cost of that flexibility: compared with a deterministic chain, you have to guard against repeated or invalid tool calls. Few-shot examples are one of the cheapest guards available. One example that shows the full sequence (user request, correct tool, correct arguments, how the result is used) can fix a path that the model otherwise gets wrong.
Which paths deserve an example? Not the obvious ones. A tool whose name and docstring already make its use clear seldom needs a demonstration. Databricks asks tool authors to write clear descriptions of the function and its arguments for exactly this reason, so the model knows how and when to call it. Spend example tokens on paths where the description is not enough. Typical cases are multi-step sequences (look up the order, then check the policy), tools whose arguments the model has to derive from the conversation, and recovery paths, such as retrying or asking the user when a tool returns an error.
Checkpoint 3 of 8· Check yourself
A help-desk agent calls lookup_order correctly for valid IDs. When the tool returns "invalid order number", the agent gives up and answers vaguely. Which candidate example adds the most?
The success path already works, so more success examples are redundant. The recovery path is the untested behaviour, and Databricks describes the desired recovery explicitly.
“the agent can retry or prompt the user for the correct ID”Source: docs.databricks.com
3.Show output structure once, and only where nothing else enforces it
The second high-value kind teaches format rather than reasoning. Models copy the shape of the examples they see. Databricks' Genie guidance says this about example SQL queries: Genie treats them as style templates and learns your formatting and coding conventions from them. Example queries are also where you show logic that metadata alone cannot explain, such as tricky calculations, specific filters and multi-step aggregations.
Format gets learned fast, which means its marginal value drops off quickly. Once one example shows the structure, a second example in the same structure teaches almost nothing new. Check whether the platform already enforces the structure. Databricks structured outputs constrain chat-model responses through a response_format field. If a JSON schema is enforced there, an example whose only job is to show the JSON shape is mostly redundant. Keep it only if it also shows something the schema cannot, such as how to fill a field when the data is missing.
response_format = {
"type": "json_object",
}Enforcing a schema is not free either. Databricks notes that the techniques behind structured outputs affect input and output token counts. Still, a schema enforces the structure on every response, while an example only demonstrates it.
Checkpoint 4 of 8· Check yourself
Your Genie space has three candidate example queries that use the same date formatting and CTE style. Two of them add no new calculation. Under a tight budget, what is the best decision?
Genie picks up style from example queries, so one example is enough to set the convention. Further slots should go to new logic.
“Genie treats example queries as style templates, learning your preferred formatting and coding conventions.”Source: www.databricks.com
Checkpoint 5 of 8· Exam question
An agent must return responses as strict JSON matching a fixed schema so a downstream pipeline can parse them without error. During evaluation, the agent occasionally returns valid but inconsistently structured free text instead of the required schema. None of the current few-shot examples show the expected output format explicitly. What is the most direct fix using few-shot example selection?
Correct answer: B — Add a few-shot example that shows a representative input paired with output in the exact required JSON structure
4.Show what to do with ambiguous inputs
Clear questions are where the model already does well. Ambiguous ones are where it guesses, so this is where an example adds the most. Databricks' RAG guidance points out that complex or ambiguous queries often do not match the terminology in the underlying data. In those cases the agent should do something deliberate: pick an interpretation using a stated rule, or, as the agent concepts page puts it, decide whether to request more input from the user.
An example that pairs an ambiguous request with the intended handling teaches a policy, not just a single answer. Genie offers a related mechanism for this: usage guidelines on example queries tell Genie when to apply a particular query, which separates similar metrics. Prefer one ambiguity example that teaches a reusable rule ("when the metric is unspecified, use net revenue and say so") over several clear-cut examples.
Checkpoint 6 of 8· Check yourself
Which candidate example is most likely to have high marginal value for an agent whose failures cluster on vague questions?
The failures are on ambiguous inputs, so the example has to show the intended handling of ambiguity. Asking for more input is a recognised agent behaviour.
“which tools to call, or whether to request more input from the user”Source: docs.databricks.com
A clear example teaches one mapping from question to answer, and the model probably had it already. An ambiguity example teaches a decision rule that applies to a whole class of underspecified questions. That is more corrected behaviour per token.
5.Check the choice: compare prompt versions on the same evaluation data
Marginal contribution is a claim you can test, not just a judgement call. Register the prompt with and without a candidate example as separate versions and run both against the same evaluation dataset. Databricks' prompt-evaluation guidance says to use consistent datasets so the comparison is fair, and to include edge cases. If removing an example does not change correctness on the cases it was meant to fix, it was not adding anything and it can go.
There is a trap here. Do not copy evaluation questions into the prompt as few-shot examples. The agent will then score well on exactly those items, and the score will no longer tell you how it generalises. Databricks recommends splitting evaluation data into training, test and validation sets to avoid overfitting. Take few-shot candidates from the training side, or from production traces, and keep the test and validation questions out of the prompt.
Checkpoint 7 of 8· Match them up
Match each candidate example to the most defensible decision under a tight token budget
Tap a term, then the definition that fits it.
Inclusion follows from new behaviour per token. Exclusion follows from redundancy or from contaminating the evaluation.
“To avoid overfitting, Databricks recommends splitting your evaluation set into training, test, and validation sets:”Source: docs.databricks.com
Checkpoint 8 of 8· Exam question
A context engineer is deciding between two candidate few-shot examples to add to a nearly full token budget, and only one can be included. Candidate X demonstrates a tool call path the agent has never exercised in any existing example. Candidate Y demonstrates a tool call path already shown clearly in two existing examples, but with slightly different wording in the user request. Which candidate has the higher marginal contribution to agent performance, and why?
Correct answer: B — Candidate X, because it is the only one of the two that covers a tool path the agent has not yet seen demonstrated
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.More few-shot examples always improve the agent, so you should fill the budget with the most representative ones.Why is that wrong?
Every example adds input tokens to every request. Examples of behaviour the agent already gets right add cost without changing outcomes. Spend the budget on difficult, diverse cases where the agent fails.
Covered in Every example is paid for on every request
2.If the output must be JSON, include several JSON examples so the model reliably follows the format.Why is that wrong?
Structured outputs can enforce a defined JSON format through response_format. Once that is in place, extra examples that only repeat the format add little. Keep an example only for things the schema cannot express.
Covered in Show output structure once, and only where nothing else enforces it
3.The best few-shot examples are the evaluation questions the agent fails, so paste them straight into the prompt.Why is that wrong?
This leaks evaluation data into the prompt and inflates scores. Keep evaluation data split, and compare prompt versions only on held-out data.
Covered in Check the choice: compare prompt versions on the same evaluation data
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/evaluate-assess-performanceOfficial docs
“Overall latency and token consumption are examples of chain performance metrics.”
↩︎ Every example is paid for on every request“Overall latency and token consumption are examples of chain performance metrics.”
↩︎ Key concept“What's the total count of tokens for LLM generations?”
↩︎ Checkpoint - 2.
“Compared to a hard-coded chain, you must guard against repeated or invalid tool calls.”
↩︎ Cover the tool paths the agent does not yet handle“the agent can retry or prompt the user for the correct ID”
↩︎ Checkpoint - 3.
“Write clear descriptions for your function and its arguments to help the LLM understand how and when to use the function.”
↩︎ Cover the tool paths the agent does not yet handle - 4.
“Doing so impacts the number of input and output tokens consumed by the model, which in turn results in billing implications.”
↩︎ Show output structure once, and only where nothing else enforces it“Structured outputs on Databricks let you generate responses in a defined JSON format as part of your AI application workflows.”
↩︎ Exam trap 2 - 5.https://www.databricks.com/blog/data-dialogue-best-practices-guide-building-high-performing-genie-spacesSecondary source
“tricky calculations, specific filters, or re-used multi-step aggregations - that metadata alone cannot explain”
↩︎ Show output structure once, and only where nothing else enforces it“Usage guidelines allow you to explicitly tell Genie when to apply a specific query.”
↩︎ Show what to do with ambiguous inputs“Genie treats example queries as style templates, learning your preferred formatting and coding conventions.”
↩︎ Checkpoint - 6.
“This can be particularly useful when dealing with complex or ambiguous queries that might not directly match the terminology used in the retrieval documents.”
↩︎ Show what to do with ambiguous inputs - 7.https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-promptsOfficial docs
“Test edge cases: Include challenging examples in your evaluation dataset.”
↩︎ Check the choice: compare prompt versions on the same evaluation data“Use consistent datasets: Evaluate all versions against the same data for fair comparison.”
↩︎ Exam trap 3 - 8.
“Compare app versions. Test different prompts, models, or app logic against the same data.”
↩︎ Check the choice: compare prompt versions on the same evaluation data
Also cited
- https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/evaluate-define-qualityOfficial docs
“The set should include difficult and diverse cases to effectively test the model's capabilities.”
↩︎ Exam trap 1“The set should include difficult and diverse cases to effectively test the model's capabilities.”
↩︎ Prediction“To avoid overfitting, Databricks recommends splitting your evaluation set into training, test, and validation sets:”
↩︎ Checkpoint - https://docs.databricks.com/aws/en/agents/conceptsOfficial docs
“which tools to call, or whether to request more input from the user”
↩︎ Checkpoint