Custom Code-Based Scorers in MLflow: Signature, Inputs and Return Values

1.What a scorer does with a trace

In the MLflow GenAI evaluation framework, scorers are the single interface for defining evaluation criteria for models, agents and applications. Every scorer follows the same three steps. It receives a Trace, from either mlflow.genai.evaluate() during development or the monitoring service in production. It parses the trace to extract the fields it needs. It runs its assessment and returns the result as Feedback attached to the trace.

The result does not have to be a number. It could be a pass/fail, true/false, numerical value, or a categorical value. Because scorers work on traces and not on a particular dataset format, you can use the same scorer to evaluate in development and to monitor in production. That keeps your evaluation consistent across the application lifecycle.

Sources1

2.Where custom scorers sit among the scorer types

Databricks describes scorer types as a ladder. Each step adds customization and complexity. The recommended path is to start with built-in judges, move to custom LLM judges for domain-specific criteria, and write code-based scorers for deterministic business logic.

Custom LLM judges let you write evaluation criteria in natural language. Code-based scorers are for cases where neither built-in nor custom LLM judges fit. The documentation names four reasons to write one:

- to define a custom heuristic or programmatic metric - to change how trace data is mapped to a built-in LLM judge - to use your own LLM in place of a Databricks-hosted judge - to get more flexibility and control than a custom LLM judge offers

Sources21

3.The @scorer signature and its four inputs

Most code-based scorers should be written as ordinary Python functions with the @scorer decorator. MLflow passes the complete trace, and it also pulls out the commonly needed parts and passes them as keyword-only named arguments:

from mlflow.genai.scorers import scorer
from typing import Optional, Any
from mlflow.entities import Feedback

@scorer
def my_custom_scorer(
    *,  # All arguments are keyword-only
    inputs: Optional[dict[str, Any]],       # App's raw input, a dictionary of input argument names and values
    outputs: Optional[Any],                 # App's raw output
    expectations: Optional[dict[str, Any]], # Ground truth, a dictionary of label names and values
    trace: Optional[mlflow.entities.Trace]  # Complete trace with all spans and metadata
) -> Union[int, float, bool, str, Feedback, List[Feedback]]:
    # Your evaluation logic here

The four inputs are:

- inputs: the request sent to the app - outputs: the app's response - expectations: ground truth or labels - trace: the full trace, passed as an instantiated mlflow.entities.trace class, so you can analyse intermediate steps, latency and tool usage

When you run mlflow.genai.evaluate(), the first three can come from the data argument or be parsed from the trace. You only declare the arguments your logic uses. A word-count metric needs nothing but outputs.

Sources3

4.Return types and how metrics get their names

For simple pass/fail or numeric results, return a primitive value. The MLflow UI shows each type differently:

@scorer
def response_length(outputs: str) -> int:
    # Return a numeric metric
    return len(outputs.split())

@scorer
def contains_citation(outputs: str) -> str:
    # Return pass/fail string
    return "yes" if "[source]" in outputs else "no"

For a detailed assessment, return a Feedback object. It carries a value plus an optional rationale, source (an AssessmentSource such as HUMAN, CODE or LLM_JUDGE) and metadata. To score several aspects in one pass, return a list of Feedback objects. Each one should set name, and each name appears as a separate metric.

Names matter because they become metric names in evaluation results, monitoring results and dashboards. Use clear names such as safety_check or relevance_monitor. The rules are:

- A primitive value or an unnamed Feedback takes the function name with @scorer, or the Scorer.name field with the class. - A named Feedback always uses its own name. - All metrics must have distinct names, and that includes each Feedback in a returned list.

Sources3

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The metric name of a @scorer is always the Python function name.Why is that wrong?

   If the scorer returns Feedback objects with name set, those names win. The function name is used only for primitive values or unnamed Feedback.

   Covered in Return types and how metrics get their names

2. 2.Custom code-based scorers can only compute heuristics and cannot involve an LLM.Why is that wrong?

   One documented use for a code-based scorer is calling your own LLM in place of a Databricks-hosted judge. Another is changing how trace data is mapped to a built-in judge.

   Covered in Where custom scorers sit among the scorer types