What you will be able to do
- Check that two prompt-configuration runs are comparable before reading their metrics
- Tell judge-scored quality metrics apart from deterministic token and latency metrics in an evaluation run
- Decide whether a higher-token prompt configuration is justified by reading per-metric deltas, not just the overall average
Key concept
Paired evaluation runs — Two prompt configurations scored against the same evaluation dataset by the same scorers, so that any change in the metrics can be put down to the prompt change and nothing else. You can only judge whether a costlier prompt is worth it when you have this pairing.
1.Making two prompt runs comparable
In this task you get experiment tracking results for two system prompt configurations and have to decide whether the more expensive one is worth its cost. Before you read any number, make sure the numbers can be compared. In MLflow tracking, a run is a single execution of code and an experiment is a collection of related runs. Inside an experiment you can compare and filter runs to see how performance changes with parameter settings, input data, and other factors. For GenAI agents, mlflow.genai.evaluate() produces these runs. It runs the agent against an evaluation dataset and applies scorers. Databricks lists "Validating prompt or model changes across app versions" as one of its main uses.
A fair comparison keeps the dataset and the scorers fixed and changes only the configuration under test. Adding challenging examples is still good practice; Databricks recommends testing edge cases. Add them to the shared dataset, then re-run both versions on it. In the Databricks evaluate-and-improve tutorial, the second evaluation run says this in its inline comments: same eval dataset, new app version, same judges.
Checkpoint 1 of 5· Check yourself
In the tutorial's evaluation of the improved app version, what is held constant relative to the first run so the comparison is fair?
The tutorial re-runs evaluation on the improved version with the same judges and the same dataset, so only the app version changes.
“Run the evaluation on the improved version using the same judges and dataset to see if you've successfully addressed the issues.”Source: docs.databricks.com
2.Two kinds of numbers: judge scores and cost metrics
An evaluation run holds two kinds of metric, and you need to keep them separate. Quality metrics come from scorers, which are often LLM judges: safety, relevance_to_query, groundedness and similar. Cost and latency metrics are counted, not judged. In Databricks' guidance on which metrics matter for an agent, the cost dimension is measured by total_token_count, total_input_token_count and total_output_token_count, and the latency dimension by latency_seconds. Both are deterministic. A longer system prompt goes out with every request, so the cost of a verbose configuration shows up in these counts, not in the judge scores.
The tutorial's comparison code deliberately keeps the two kinds apart. When it builds the version-comparison table, it filters out every metric column whose name contains agent/. Its own comment says these are the latency and token-count metrics:
# Extract metric columns (they end with /mean, not .aggregate_score)
# Skip the agent metrics (latency, token counts) for quality comparison
metric_cols = [col for col in run_v1_df.columns
if col.startswith('metrics.') and col.endswith('/mean')
and 'agent/' not in col]Checkpoint 2 of 5· Check yourself
In the tutorial's comparison code, what does the condition 'agent/' not in col remove from the quality comparison?
The code comment says the agent metrics (latency and token counts) are skipped so the comparison table and average cover quality alone. You have to read cost separately.
“Skip the agent metrics (latency, token counts) for quality comparison”Source: docs.databricks.com
3.Reading the comparison: is the costlier version justified?
| Metric | V1 Score | V2 Score | Improvement |
|---|---|---|---|
| safety | 1.000 | 1.000 | +0.000 |
| professional_tone | 1.000 | 1.000 | +0.000 |
| follows_instructions | 0.571 | 0.714 | +0.143 |
| includes_next_steps | 0.286 | 0.571 | +0.286 |
| mentions_contact_name | 1.000 | 1.000 | +0.000 |
| retrieval_groundedness | 0.857 | 0.571 | -0.286 |
| concise_communication | 0.286 | 1.000 | +0.714 |
| relevance_to_query | 0.714 | 1.000 | +0.286 |
The tutorial prints "Overall average improvement: +0.143 (+20.0%)". That one number hides how the gains are spread. Most of the improvement comes from concise_communication (+0.714). Three metrics were already at 1.000 and stayed there. retrieval_groundedness fell from 0.857 to 0.571, so v2 makes more claims that aren't backed by retrieved context. Also note that the code's Improved column marks a metric ✓ whenever the change is zero or better. A row of ticks is not a row of gains.
Deciding whether a higher-token configuration is justified means reading three things together: the quality metrics that rose, any metric that fell, and the token and latency metrics the quality table left out. The Databricks sources give you these measurements but no threshold for how much quality gain per extra token is enough. That call is yours, and it should account for which metrics matter most to the agent's job. A gain in concision can be worth paying for. A new groundedness failure is harder to accept, because it means the agent says things its sources don't support.
Checkpoint 3 of 5· Check yourself
Which statement about the tutorial's "Overall average improvement: +0.143 (+20.0%)" is accurate?
The code calculates the overall figure from quality metrics only. It contains no cost, and as a plain average it absorbs the -0.286 groundedness drop.
“Calculate overall improvement (only for quality metrics)”Source: docs.databricks.com
Checkpoint 4 of 5· Exam question
MLflow experiment tracking logs two configurations of a support agent handling 10,000 queries per day. Config A costs $0.002 per query and resolves 70% of queries successfully. Config B costs $0.005 per query and resolves 92% of queries successfully. Using cost per successful resolution as the deciding metric, which configuration is more cost-effective?
Correct answer: B — Config A, because its per-query cost is lower and per-query cost is what determines cost-effectiveness
Sources5
4.Confirming the cost side in production
Token counts from an offline evaluation describe the evaluation dataset, not live traffic. On Databricks, Unity Gateway usage tracking logs model requests to the system.ai_gateway.usage system table, including token usage and latency. You can query it from Databricks SQL or a notebook to track costs per endpoint. Databricks also recommends that you keep evaluating prompts after deployment to catch degradation. Together, these let you check that the cost and quality tradeoff you accepted offline still holds in production: compare live token usage and latency against the offline counts, and re-score quality on live traffic.
Checkpoint 5 of 5· Check yourself
Where does Unity Gateway log request-level usage data, such as token counts and latency, for model requests in production?
Unity Gateway writes usage data, including token usage and latency for model requests, to the system.ai_gateway.usage table, which you can query from Databricks SQL or a notebook.
“Unity Gateway logs usage data to the system.ai_gateway.usage system table.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A positive overall average improvement shows that the higher-token prompt version is justified.Why is that wrong?
The tutorial's overall figure averages quality metrics only. It hides per-metric regressions, such as the retrieval_groundedness drop, and leaves out token cost entirely.
Covered in Reading the comparison: is the costlier version justified?
2.You can compare a new prompt version, evaluated on a refreshed dataset, against the old version's earlier run.Why is that wrong?
If the dataset changes, a metric difference can't be attributed to the prompt. Evaluate every version against the same data and the same scorers.
Covered in Making two prompt runs comparable
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/mlflow/trackingOfficial docs
“you can compare and filter runs to understand how your model performs and how its performance depends on the parameter settings”
↩︎ Making two prompt runs comparable - 2.
“Validating prompt or model changes across app versions”
↩︎ Making two prompt runs comparable - 3.https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-promptsOfficial docs
“Test edge cases: Include challenging examples in your evaluation dataset.”
↩︎ Making two prompt runs comparable“Monitor production: Continue evaluating prompts after deployment to catch degradation.”
↩︎ Confirming the cost side in production“Use consistent datasets: Evaluate all versions against the same data for fair comparison.”
↩︎ Key concept“Use consistent datasets: Evaluate all versions against the same data for fair comparison.”
↩︎ Exam trap 2“Use consistent datasets: Evaluate all versions against the same data for fair comparison.”
↩︎ Prediction - 4.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/evaluate-assess-performanceOfficial docs
“Below is a summary of the metrics that Databricks recommends for measuring the quality, cost, and latency of your RAG application.”
↩︎ Two kinds of numbers: judge scores and cost metrics“What's the total count of tokens for LLM generations?”
↩︎ Two kinds of numbers: judge scores and cost metrics - 5.
“Skip the agent metrics (latency, token counts) for quality comparison”
↩︎ Two kinds of numbers: judge scores and cost metrics“This comparison view is the primary way to verify that a new prompt version outperforms the previous one.”
↩︎ Reading the comparison: is the costlier version justified?“Overall average improvement: +0.143 (+20.0%)”
↩︎ Reading the comparison: is the costlier version justified?“Calculate overall improvement (only for quality metrics)”
↩︎ Exam trap 1“Run the evaluation on the improved version using the same judges and dataset to see if you've successfully addressed the issues.”
↩︎ Checkpoint“Calculate overall improvement (only for quality metrics)”
↩︎ Checkpoint - 6.
“For model requests, it records metrics such as token usage and latency.”
↩︎ Confirming the cost side in production“Unity Gateway logs usage data to the system.ai_gateway.usage system table.”
↩︎ Checkpoint