Comparing GenAI App Versions with MLflow Evaluation Runs

1.Why model selection starts with an evaluation run

Choosing the "best" model for a GenAI task on Databricks comes down to metrics that your own experiments produced, on your own task. In MLflow 3 those metrics come from an evaluation run: the result of calling mlflow.genai.evaluate() with an evaluation dataset and a list of scorers.

An evaluation run does three things. It runs your app on every input in the dataset and captures a trace for each one. It applies each scorer to every trace, which produces feedback assessments. It then stores aggregated pass rates and metrics alongside the individual traces. You get one summary number per scorer, and behind each number are the exact traces that produced it.

Each scorer becomes a metric you can compare. The built-in LLM judges are pre-built evaluators for common dimensions such as correctness, relevance, safety, groundedness and guideline adherence. Custom LLM judges apply your own rubric. Code-based scorers are deterministic Python checks, for example exact match, format validation, latency checks or business rules.

Model selection then works like this: each candidate gets its own evaluation run, whether it's a different LLM, a different prompt version or a different retrieval setup. Choosing between candidates means comparing those runs. Evaluation runs are a special type of MLflow Run, so you can compare them in the UI and also query them programmatically.

Which metrics matter depends on what the task needs. Databricks groups the common metrics for a RAG application into retrieval quality, response quality, and system performance (cost and latency). Response metrics cover whether the answer is correct, relevant, grounded in the retrieved context (not a hallucination) and safe. Cost and latency are computed deterministically from the app's outputs, as token counts and execution time. Collect both retrieval and response metrics, because an app can respond poorly despite retrieving the correct context, and the reverse. The table also shows which metrics need ground truth, meaning a known-good answer or document list in your evaluation set.

Sources12

2.Change one thing: same dataset, same judges

Comparing two runs only tells you something if the runs differ in the one thing you are choosing between. In the Databricks tutorial, the team improves an email-generation app and then reruns evaluation on the new version "using the same judges and dataset". The function being evaluated is passed as predict_fn, while the dataset and the scorers stay fixed.

Be careful about what changes inside that function. The tutorial's v2 function rewrites the prompt and also calls databricks-claude-sonnet-4-5, so that comparison bundles changes together. That is fine for asking whether the new app version is better overall, but it cannot tell you whether the prompt or the LLM caused the difference. To pick between LLMs, vary only the model and hold the prompt fixed.

The prompt-evaluation guide states the fairness rule as a best practice: evaluate every version against the same data. Suppose v2 scores higher on a dataset that is easier, or with a looser judge. That says nothing about whether v2 is the better model. Wrapping the call in mlflow.start_run(run_name=...) gives each candidate a readable name in the UI, which you will need when you pick runs to compare.

with mlflow.start_run(run_name="v2"):
    eval_results_v2 = mlflow.genai.evaluate(
        data=eval_dataset, # same eval dataset
        predict_fn=generate_sales_email_v2, # new app version
         ? =email_judges,

Sources34

3.Reading a side-by-side comparison

In the MLflow UI you compare evaluation runs from the experiment. Open the experiment, click Evaluation runs in the left sidebar, tick the runs you want to compare, and choose Compare from the Actions menu. The right pane then compares each trace across the selected runs. Click a request identifier to see that request's full traces from each run, and click See details to see each assessment. The tutorial calls this comparison view the primary way to verify that a new prompt version outperforms the previous one.

Quality scores are only one part of the choice. Latency and token counts (a proxy for cost) are separate metrics, and requirements often fix a limit on them. Databricks notes that most production applications have a latency budget, and that low-latency, real-time use cases differ from high-throughput batch ones. A practical way to combine the two kinds of evidence is to treat the latency or token budget as a requirement first. Then compare the quality scores of the candidates that meet it, and check those candidates for per-metric regressions. A candidate that is slightly better on quality but outside the budget does not satisfy the task.

Sources351

4.Comparing evaluation metrics in code

Evaluation runs are MLflow runs, so mlflow.search_runs() returns their metrics as DataFrame columns. The tutorial fetches each run by its run_id in a separate call, because search_runs doesn't support IN or OR operators. It then keeps only the quality metrics: columns that start with metrics. and end with /mean.

import pandas as pd

# Fetch runs separately since mlflow.search_runs doesn't support IN or OR operators
run_v1_df = mlflow.search_runs(
    filter_string=f"run_id = '{eval_results_v1.run_id}'"
)
run_v2_df = mlflow.search_runs(
    filter_string=f"run_id = '{eval_results_v2.run_id}'"
)

# Extract metric columns (they end with /mean, not .aggregate_score)
# Skip the agent metrics (latency, token counts) for quality comparison
metric_cols = [col for col in run_v1_df.columns
               if col.startswith('metrics.') and col.endswith('/mean')
               and 'agent/' not in col]

Notice the 'agent/' not in col filter. The evaluation run also records agent metrics such as latency and token counts. They are real experiment metrics, but they measure a different thing from the judge scores. The tutorial leaves them out of the quality average so that, for example, a faster version doesn't look "higher quality" because of its speed. When cost or speed matters for the choice, read those agent metrics as a separate criterion, for example against a latency or token budget, rather than folding them into one quality number.

Not every scorer can run on any dataset. An evaluation dataset has inputs, and optionally expectations: the correct output, used by scorers that need ground truth. Judges such as Correctness compare the agent's response against a known-good answer, so they need expectations. Reference-free judges such as safety, groundedness and relevance to the query do not need a known-good answer, so they can run on inputs alone. Before you plan a model comparison, check that your dataset carries the expectations your chosen metrics need.

Sources312

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Two candidates' scores can be compared directly even if each was evaluated on its own dataset or with its own judges.Why is that wrong?

   A comparison is only fair when every version is evaluated against the same data with the same scorers. Only the candidate itself should change.

   Covered in Change one thing: same dataset, same judges

2. 2.If the new version's overall average score is higher, it is better on every dimension and can be chosen without further checks.Why is that wrong?

   An average can hide a regression. In the tutorial, retrieval_groundedness dropped even though the overall average rose 20%. The comparison exists to catch regressions as well as improvements.

   Covered in Reading a side-by-side comparison