GenAI Evaluation Phase: Scoring an Agent Before Release

1.Two phases of one quality loop

The exam asks you to tell apart two phases of quality work on a Gen AI application. The Databricks RAG cookbook separates them by time: evaluation happens during development, and monitoring happens once the application is deployed to production. The same sentence adds that the fundamental components are similar. Both phases need a definition of quality, a way to score responses against it, and traces that show how each response was produced. The differences are when the scoring runs, what data it runs over, and what you do with the result.

MLflow treats the two phases as stages of one continuous loop, not as a hand-off from one team to another. You trace the agent and find an issue. You collect feedback and curate a dataset, then write a scorer that catches the issue automatically. You fix the agent and re-run evaluation to confirm the fix. Then you monitor production with the same scorers so the issue doesn't come back. Monitoring then surfaces the next issue, and the loop starts again. Evaluation sits in the middle of the loop and monitoring at its end. The scorer is what carries over from one to the other.

Sources12

2.What evaluation measures: scorers and judges

Evaluation answers one question: is the agent actually good? In the documentation's words, is the response correct, grounded, safe, and complete? MLflow answers it with scorers, which are LLM judges and code-based checks that attach a quality assessment to each trace. A scorer can return pass/fail, true/false, a number, or a category.

LLM judges are central here because Gen AI output is open-ended. The RAG cookbook points out that reading every response on every evaluation run isn't feasible. A separate LLM reviewing the outputs scales the work and can check things human raters struggle with at volume, such as whether a response is grounded in thousands of tokens of retrieved context. MLflow lets you start simple and add control as your needs grow:

The way a scorer works is what links the two phases. It receives a trace, parses out the fields it needs, runs its assessment, and returns the result as Feedback that is attached to the trace. The scorer doesn't care whether the trace came from an evaluation run or from the monitoring service. That's why one scorer can serve both phases.

Sources314

3.Running the evaluation phase with mlflow.genai.evaluate()

In development, evaluation runs against an evaluation set: a curated set of queries (ideally with expected outputs) that represents how the application will be used. The cookbook says these examples should be challenging, diverse, and updated as usage changes. An evaluation harness then runs the application on every record and passes each output through the judges. In MLflow that harness is mlflow.genai.evaluate(). You can start by scoring a few traces from the UI, then move the same scorers into a repeatable evaluate() loop as the app matures.

def mlflow.genai.evaluate(
    data: Union[pd.DataFrame, List[Dict], mlflow.genai.datasets.EvaluationDataset],  # Test data.
    scorers: list[mlflow.genai.scorers.Scorer],  # Quality metrics, built-in or custom.
    predict_fn: Optional[Callable[..., Any]] = None,  # App wrapper. Used for direct evaluation only.
    model_id: Optional[str] = None,  # Optional version tracking.
) -> mlflow.models.evaluation.base.EvaluationResult:

The documentation lists three typical occasions for evaluation: nightly or weekly checks against curated datasets, validating prompt or model changes across app versions, and checks before a release or PR to prevent quality regressions. All three are deliberate, on-demand runs that compare one version against another. Evaluation is how you choose a version before users see it.

Beyond calling a predict_fn, evaluate() has an "answer sheet" mode for cases where you can't, or don't want to, run the agent during evaluation. You supply inputs and pre-computed outputs, or existing traces, and evaluate() runs the scorers and logs an evaluation run. The example below pulls traces from production and scores them in a batch. This is still evaluation: it is a run you trigger, over a fixed set of traces, and it produces an evaluation result.

import mlflow

# Retrieve traces from production
traces = mlflow.search_traces(
    filter_string="trace.status = 'OK'",
)

# Evaluate problematic traces
evaluation = mlflow.genai.evaluate(
    data=traces,
    scorers=[Safety(), RelevanceToQuery()]
)

Answer-sheet mode has one catch. If the answer sheet contains traces that differ from what your production environment emits, a scorer that parses them may not work on live traces. The documentation warns you may need to rewrite your scorer functions before using them for production monitoring. Using your production app as the predict_fn avoids that mismatch.

Sources135

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Using mlflow.genai.evaluate() on traces pulled from production makes it production monitoring.Why is that wrong?

   evaluate() can score existing traces, including production ones, but it is still an on-demand batch evaluation run. Monitoring is the continuous, background scoring of sampled live traces by scheduled scorers.

   Covered in Running the evaluation phase with mlflow.genai.evaluate()

2. 2.A scorer written against an answer sheet always works unchanged in production monitoring.Why is that wrong?

   If the answer-sheet traces differ from what production emits, the scorer's parsing may not match live traces and may need rewriting.

   Covered in Running the evaluation phase with mlflow.genai.evaluate()