What you will be able to do
- Trace an aggregate metric regression down to the specific responses and judge rationales that caused it
- Break a prompt version diff into separate elements and connect each one to the metrics it should move
- Use prompt lineage, logged parameters and the Compare Runs view to confirm which configuration change drives a tradeoff
- Name the specific v2 prompt guideline behind the includes_next_steps gain and the retrieval_groundedness regression in the sales-email tutorial
1.From aggregate deltas to the regressing traces
To find the element behind a tradeoff, you have to go down from metrics to traces. The tutorial pulls the traces from both evaluation runs, builds a merge key from each request's input parameters so the same request is lined up across versions, and flags every judge whose verdict changed from yes to no.
# Get detailed traces for both versions
traces_v1 = mlflow.search_traces(run_id=eval_results_v1.run_id)
traces_v2 = mlflow.search_traces(run_id=eval_results_v2.run_id)The output reports 7 matching examples and 2 regressions, both on retrieval_groundedness. In the TechStart example, both versions list the same three tickets. The v1 email stays with facts the retrieved context confirms. The v2 email adds a promise: "I'll contact our support team today and provide you with a detailed status update by end of day tomorrow." The judge's rationale picks out exactly that commitment as unsupported. The judge rationale is the most precise evidence you have, because it names the kind of sentence the new configuration now produces. Your job is to find the instruction in the prompt diff that would produce that kind of sentence. The tutorial does show the v2 prompt, and guideline 3 is the match: "End with a specific, actionable next step that includes a concrete timeline", with examples such as "I'll follow up with pricing by Friday". That one instruction explains both sides of the tradeoff. It asks for a next step with a timeline, which fits the includes_next_steps gain (+0.286). It also pushes the agent to write a dated commitment that nothing in the retrieved customer context supports, which fits the retrieval_groundedness drop (-0.286). The tutorial does not run an ablation, so treat this as the best-supported candidate and confirm it by changing only that guideline and re-running the comparison.
Checkpoint 1 of 6· Check yourself
In the TechStart regression, why did the judge fail v2 on retrieval_groundedness?
The v2 rationale confirms the tickets match the retrieved context. It flags only the added promise of a status update as unsupported.
“which is not directly supported by the retrieved context”Source: docs.databricks.com
Checkpoint 2 of 6· Check yourself
Which element of the v2 sales-email prompt best explains both the includes_next_steps gain and the unsupported promise in the TechStart email?
Guideline 3 asks for a next step with a concrete timeline. That would raise includes_next_steps, and it also invites a dated commitment the retrieved context cannot support.
“End with a specific, actionable next step that includes a concrete timeline”Source: docs.databricks.com
Checkpoint 3 of 6· Exam question
A team is deciding whether MLflow experiment tracking results justify adopting a higher-token system prompt configuration for a production agent. Which factors should they weigh in that decision? (Select 3)(Select 3)
Correct answers: A, B, C — The measured quality or accuracy delta reported by the scorers or LLM judges for each configuration; The incremental latency and per-request cost the added tokens introduce at production query volume; The business criticality and frequency of the task the agent performs
Sources1
2.Breaking a prompt diff into elements
A higher-token configuration is rarely one change. It's usually several instructions bundled together. The Databricks prompt-evaluation guide shows this clearly. Version 1 is the one-line template Summarize this text: {{content}}. Version 2 replaces it with a much longer template. Every element in that template is sent on every call, so each one adds to the input token count. Each element is also aimed at a different behaviour.
| Element | Text in v2 template |
|---|---|
| Role line | You are an expert summarizer. |
| Exact length constraint | exactly 2 sentences (no more, no less - be very careful about the number of sentences) |
| Fact-coverage guideline | Include ALL core facts and key findings |
| Style guidelines | Use clear, concise language; Write for a general audience |
| Repeated constraint | Use exactly 2 sentences |
That summarization example has no reported metrics, so it only shows how to split a diff. The sales-email tutorial gives you both the prompt and the metric deltas. Its v2 prompt has a system message, a MOST IMPORTANT block that puts the user instructions first, and four numbered guidelines.
| Element | Text in v2 prompt | Metric it should move and the reported change |
|---|---|---|
| System message | You are a helpful sales assistant who writes concise, instruction-focused emails. | concise_communication +0.714 |
| MOST IMPORTANT block | MOST IMPORTANT: Follow these specific user instructions exactly: | follows_instructions +0.143 |
| Guideline 1 | PRIORITIZE the user instructions above all else | follows_instructions +0.143 |
| Guideline 2 | Keep the email CONCISE - only include information directly relevant to the user's request | concise_communication +0.714 |
| Guideline 3 | End with a specific, actionable next step that includes a concrete timeline | includes_next_steps +0.286, but retrieval_groundedness -0.286 |
| Guideline 4 | Only reference customer information if it's directly relevant to the user's instructions | retrieval_groundedness |
Matching each element to the scorer it should move is a working method for reading a diff, not a procedure that Databricks defines. A concision instruction should show up in a concision judge, and a next-step instruction in a next-step check. With that mapping, three outcomes are possible. An element whose target metric didn't move is cost with no benefit and is a candidate for removal. An element whose target improved is earning its tokens. An element whose target improved while another metric regressed is the one driving the tradeoff, and you keep, rewrite or drop it depending on which metric matters more. In the table, guideline 3 is the only element tied to a gain and a regression, which is why it is the one to test. The registry also helps here: each version carries a commit message, and tags can say what the change was meant to do.
updated_prompt = mlflow.genai.register_prompt(
name=f"{uc_schema}.{prompt_name}",
template=improved_template,
commit_message="Added structured response guidelines for better customer experience",
tags={
"author": "support-team@company.com",
"improvement": "Added empathy guidelines and response structure"
}
)Checkpoint 4 of 6· Check yourself
Months later, a reviewer needs to know why the long guideline block was added to a prompt version. Which practice does Databricks recommend for recording that?
Databricks' prompt-evaluation best practices say to use meaningful commit messages to record why each change was made, which is what commit_message in register_prompt is for.
“Document changes: Use meaningful commit messages to track why changes were made.”Source: docs.databricks.com
3.Linking prompt versions to runs and parameters
Before you blame the prompt text for a cost difference, confirm that the prompt is the only thing that changed. If an application sets an active model and loads its prompt from the registry, MLflow automatically records which prompt version each application version used. You can see this on the experiment's Versions tab. The same tutorial logs configuration parameters (llm, temperature, max_tokens) against the application version. Each of these also affects the token counts and quality metrics. If two configurations differ in model or max_tokens as well as prompt, you can't put the token difference down to the prompt alone.
The Compare Runs page has a toggle that hides parameters and metrics that are identical across the selected runs, so only the differences remain. When you have many variant runs, a parallel coordinates chart can show which parameter values go with which metric values. Grouping runs by a parameter value does the same job in the chart view.
Checkpoint 5 of 6· Check yourself
You open Compare Runs for two prompt configurations and see dozens of logged parameters. What is the quickest way to check whether anything besides the prompt version differs?
Hiding identical values leaves only the parameters that actually differ, which shows at once whether something like max_tokens changed along with the prompt.
“you can hide parameters and metrics that are identical in all selected runs by toggling”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
An experiment tracking dashboard shows quality score rising sharply as the first three few-shot examples are added to a system prompt, then flattening: a tenth near-duplicate example added after that raises the token count substantially but leaves the quality score essentially unchanged. What should the context engineer conclude is driving the cost-performance tradeoff, and what should they do?
Correct answer: A — The near-duplicate examples added past the point of diminishing returns are inflating tokens without adding quality, so they should be trimmed
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.The per-metric comparison table is enough to identify which prompt element caused a regression.Why is that wrong?
Aggregate deltas show which metric moved, not why. You have to go to the specific examples that regressed and read the judge rationales to see what behaviour the new prompt introduced.
2.If the higher-token configuration has a longer system prompt, the extra prompt text must explain all of its extra token cost.Why is that wrong?
Application versions also log parameters such as the LLM, temperature and max_tokens. Check that these are identical across the runs before attributing the difference to the prompt.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Look for specific examples where the evaluation metrics regressed so you can focus on those.”
↩︎ From aggregate deltas to the regressing traces“which is not directly supported by the retrieved context”
↩︎ From aggregate deltas to the regressing traces“Found 7 matching examples between v1 and v2”
↩︎ From aggregate deltas to the regressing traces“End with a specific, actionable next step that includes a concrete timeline”
↩︎ From aggregate deltas to the regressing traces“PRIORITIZE the user instructions above all else”
↩︎ Breaking a prompt diff into elements“Look for specific examples where the evaluation metrics regressed so you can focus on those.”
↩︎ Exam trap 1 - 2.https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-promptsOfficial docs
“Register different prompt versions representing different approaches to your task:”
↩︎ Breaking a prompt diff into elements“Document changes: Use meaningful commit messages to track why changes were made.”
↩︎ Breaking a prompt diff into elements - 3.https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry/track-prompts-app-versionsOfficial docs
“MLflow automatically tracks the relationship between the prompt version and the application version.”
↩︎ Linking prompt versions to runs and parameters“These parameters help you track the configuration of this app version”
↩︎ Exam trap 2 - 4.https://docs.databricks.com/aws/en/mlflow/runsOfficial docs
“you can hide parameters and metrics that are identical in all selected runs by toggling”
↩︎ Linking prompt versions to runs and parameters - 5.
“A parallel coordinates plot is useful in understanding the effect of parameter settings on model performance and investigating relationships between parameters and metrics.”
↩︎ Linking prompt versions to runs and parameters