Setting Up MLflow 3 Evaluations to Compare Agent Context Configurations

1.Why context configurations are compared with mlflow.genai.evaluate()

A context configuration covers everything you decide about what reaches the model: the prompt template, what gets retrieved and how much, and generation settings such as temperature and max tokens. Change any of these and you have a new version of your agent. The question this objective asks is always the same: across several runs, which version produced the most reliable results on a given task? Reading a handful of outputs by eye can't answer that. MLflow 3 replaces it with a structured harness. You don't have to run the agent and check each output yourself. MLflow Evaluation takes in test data, runs your agent and scores the results automatically. The docs say this "makes it easier to compare versions, track improvements, and share results across teams." They also list "validating prompt or model changes across app versions" as a core use case, and that is exactly what comparing context configurations is.

def mlflow.genai.evaluate(
    data: Union[pd.DataFrame, List[Dict], mlflow.genai.datasets.EvaluationDataset],  # Test data.
    scorers: list[mlflow.genai.scorers.Scorer],  # Quality metrics, built-in or custom.
    predict_fn: Optional[Callable[..., Any]] = None,  # App wrapper. Used for direct evaluation only.
    model_id: Optional[str] = None,  # Optional version tracking.
) -> mlflow.models.evaluation.base.EvaluationResult:

Each parameter plays its own part in a comparison. data and scorers are the measuring stick, so they stay the same from run to run. predict_fn is the configuration being measured, and model_id is an optional label that links the run to a tracked version. The function returns an EvaluationResult, and the results are stored as traces with scorer feedback in the active MLflow experiment. Every configuration you test therefore leaves a record you can come back to and compare.

Sources1

2.Keep the dataset and scorers the same; change only the app

The Databricks guide to comparing prompt versions spells out the rule: evaluate every version against the same data. It also says to "include challenging examples in your evaluation dataset", because two configurations that both handle the easy cases may behave differently on the hard ones. Each record in the dataset has a required inputs dictionary, which is passed to your app, and an optional expectations dictionary, which holds ground truth for the scorers. In the prompt-comparison example, expectations contains a list of expected_facts that a good summary should include. Ground truth like this lets a scorer check correctness and completeness, not just fluency.

The "Evaluate and improve an agent" tutorial shows the pattern in code. The second version of the app is evaluated inside a named run, using the same dataset and the same judges as the first version. The inline comments say this directly. Naming the run with run_name makes it easy to find in the UI when you compare runs later.

with mlflow.start_run(run_name="v2"):
    eval_results_v2 = mlflow.genai.evaluate(
        data=eval_dataset, # same eval dataset
         ? =generate_sales_email_v2, # new app version
        scorers=email_judges, # same judges as step 4
    )

Sources21

3.Record which configuration each run used

A score is only useful if you know which configuration produced it. MLflow 3 records this with mlflow.set_active_model(), which creates a LoggedModel for the current version of your application. In the docs' words, this LoggedModel "doesn't store your actual application code - instead, it acts as a central record" that links to things like a Git commit. You attach the configuration settings with mlflow.log_model_params(). In the tutorial these settings are the LLM, a temperature of 0.7 and max_tokens of 500, and the comment above them says they "help you track the configuration of this app version." When the agent loads its prompt from the MLflow Prompt Registry (currently in Beta), MLflow also links the prompt version to the application version. That link appears under the experiment's Versions tab.

Sources3

4.Choose scorers and an evaluation mode that carry over to production

Scorers decide what "reliable" means for your task. A scorer can return "a pass/fail, true/false, numerical value, or a categorical value." Databricks recommends a progression, beginning with this advice: "Start with built-in judges for quick evaluation." When those stop being enough, you move to custom judges or code-based scorers. You can also score existing traces from the UI. In the Traces tab, select the traces, choose Actions, then Evaluate, and pick judges such as Correctness, Completeness or Fluency. MLflow attaches each result to its trace and records the run under the Evaluation runs tab.

You also choose how outputs are produced. Direct evaluation is the recommended mode. MLflow calls your predict_fn (or a Model Serving endpoint wrapped with to_predict_fn) and captures fresh traces. Because of this, the docs say, "this mode enables you to reuse the scorers defined for offline evaluation in production monitoring since the resulting traces will be identical." Answer-sheet evaluation is for cases where you can't run the agent, or don't want to. For example, you might already have outputs from historical traces or batch jobs. In that mode you supply inputs with pre-computed outputs, or existing trace objects, and MLflow only runs the scorers.

Sources41

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.It's fine to give each context configuration a slightly different evaluation dataset, as long as the scorers are the same.Why is that wrong?

   A comparison is only fair if every version is evaluated against the same data. If the dataset changes, a difference in scores can't be attributed to the configuration.

   Covered in Keep the dataset and scorers the same; change only the app

2. 2.Scorers that worked on answer-sheet traces will always work unchanged for production monitoring.Why is that wrong?

   Only direct evaluation guarantees traces identical to production. Answer-sheet traces may differ, and the scorers may then need to be rewritten.

   Covered in Choose scorers and an evaluation mode that carry over to production