Building and Running Custom Scorers: Trace Patterns, Errors and Production Monitoring

1.Iterating on a scorer against stored traces

A code-based scorer is a Python function decorated with @scorer that receives an MLflow trace and returns a value or Feedback. To get it right you usually need several attempts, and calling an LLM-backed app on every attempt is slow. Databricks recommends a workflow that runs the app once and reuses its traces:

1. Define evaluation data. 2. Generate traces from your app. 3. Query and store the resulting traces. 4. Iterate on the scorer against the stored traces.

After that run, mlflow.search_traces(run_id=eval_results.run_id) returns the traces as a Pandas DataFrame. You pass that DataFrame straight back to evaluate() as data and leave out predict_fn. Only the new scorer runs, and the app is not called.

from mlflow.genai.scorers import scorer

@scorer
def response_length(outputs: str) -> int:
    # Example metric.
    # Implement your actual metric logic here.
    return len(outputs)

# Note the lack of a predict_fn parameter.
mlflow.genai.evaluate(
    data=generated_traces,
    scorers=[response_length],
)

Sources1

2.Two common patterns: reading spans and wrapping a judge

When the scorer declares the trace argument, it can read individual spans with their inputs, outputs, attributes and timing. The documented example finds the chat-model span and checks that the LLM call finished within 5 seconds. It returns "yes" or "no" as Feedback, with a rationale that states the measured time.

@scorer
def llm_response_time_good(trace: Trace) -> Feedback:
    # Search particular span type from the trace
    llm_span = trace. ? (span_type=SpanType.CHAT_MODEL)[0]

    response_time = (llm_span.end_time_ns - llm_span.start_time_ns) / 1e9 # convert to seconds

The second pattern wraps a built-in LLM judge. Use it to preprocess trace data for the judge or to post-process the judge's feedback. In the example, the app's inputs is a whole chat history: {"messages": [...]}. The relevance judge needs a single request, so the scorer walks the messages backwards to find the last user message. It then calls is_context_relevant and returns that judge's Feedback unchanged.

    if not last_user_message_content:
        raise Exception("Could not extract the last user message from inputs to evaluate relevance.")

    # Call the `relevance_to_query judge. It will return a Feedback object.
    return is_context_relevant(
        request=last_user_message_content,
        context={"response": outputs},
    )

Sources2

3.Expectations: available offline, missing in production

Expectations are ground-truth values or labels. When you run mlflow.genai.evaluate(), you can supply them in two ways. One is an expectations key on each row of a list or DataFrame. The other is a trace field, as in the DataFrame from mlflow.search_traces(), which carries any Expectation data attached to the traces.

Production monitoring works differently. Registered scorers always parse inputs and outputs from the trace, and expectations is not available, because live traffic has no ground truth. A scorer that you reuse in both places must not assume that expectations is filled in.

Sources2

4.When a scorer throws

If a scorer fails on one trace, MLflow records the error for that trace and carries on with the others. The recommended approach is to let exceptions propagate. MLflow catches the exception and creates a Feedback whose value is None and whose error holds the exception details, and you can open that row in the results to see them. When you want your own message, catch the exception and return a Feedback with error set. That can be the exception object itself, or an AssessmentError with a structured error_code.

        if missing:
            return Feedback(
                error=AssessmentError(
                    error_code="MISSING_REQUIRED_FIELDS",
                    error_message=f"Missing required fields: {missing}",
                ),
            )

        return Feedback(
            value=True,
            rationale="Valid JSON with all required fields"
        )

    except json.JSONDecodeError as e:
        return Feedback(error=e)  # Can pass exception object directly to the error parameter

Sources3

5.@scorer vs the Scorer class, and running scorers in production

The Scorer class is a Pydantic object. You must set its name field, add any extra fields you need, and put your logic in __call__. Keep state in instance attributes, not mutable class attributes: a class-level results = [] is shared by every instance.

# CORRECT: Use instance attributes
class GoodScorer(Scorer):
    results: list[str] = None

    name: str = "good_scorer"

    def __init__(self):
        self.results = []  # Per-instance state

    def __call__(self, outputs, **kwargs):
        self.results.append(outputs)  # Safe
        return Feedback(value=True)

Production monitoring accepts built-in LLM judges and @scorer functions. It does not accept Scorer subclasses. If you need state in production, keep it inside the body of a decorated function. The function must also be defined and registered from a Databricks notebook, because the monitoring service serializes its code to run it remotely.

Scorers that call an external LLM endpoint can read credentials from Databricks secrets, both in development and in production. dbutils is not available in the scorer runtime by default, so import it inside the function.

@scorer
def custom_llm_scorer(trace: Trace) -> Feedback:
    # Explicitly import dbutils to access secrets
    from  ?  import dbutils

    # Retrieve your API key from Databricks secrets
    api_key = dbutils.secrets.get(scope='my-scope', key='api-key')

To deploy a decorated scorer, register it and then start it with a sampling configuration. A sample_rate of 1 scores every trace.

# Register and start the scorer
custom_llm_scorer.register()
custom_llm_scorer.start(sampling_config = ScorerSamplingConfig(sample_rate=1))

Sources43

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Any custom scorer that works in mlflow.genai.evaluate() can be registered for production monitoring.Why is that wrong?

   Production monitoring supports built-in judges and @scorer functions registered from a Databricks notebook. Scorer subclasses are not supported.

   Covered in @scorer vs the Scorer class, and running scorers in production

2. 2.A scorer that depends on expectations works the same way in production monitoring as in offline evaluation.Why is that wrong?

   Live traffic has no ground truth. Registered production scorers receive inputs and outputs only, so a reused scorer has to cope without expectations.

   Covered in Expectations: available offline, missing in production

3. 3.dbutils is available inside a scorer just as it is in a notebook cell, so secrets can be read directly.Why is that wrong?

   The scorer runtime has no dbutils by default. You must import it from databricks.sdk.runtime inside the scorer function.

   Covered in @scorer vs the Scorer class, and running scorers in production

Practise it for real

Develop a custom scorer against stored traces without rerunning your app on every change

1. 1.Create an eval_dataset list whose rows each have an "inputs" dict, and a traced app function (decorated with @mlflow.trace)

   Why: Evaluation needs input rows, and tracing records what the scorer will read

   You should see: A list of rows plus a callable app

2. 2.Run mlflow.genai.evaluate(data=eval_dataset, predict_fn=sample_app, scorers=[placeholder_metric]), where placeholder_metric is a @scorer that returns 1

   Why: evaluate() needs at least one scorer, and this run exists only to produce traces

   You should see: One trace in the experiment for each dataset row

3. 3.Store the traces with generated_traces = mlflow.search_traces(run_id=eval_results.run_id)

   Why: The stored traces become reusable evaluation input

   You should see: A Pandas DataFrame of traces

4. 4.Write a real @scorer, for example response_length on outputs, and run mlflow.genai.evaluate(data=generated_traces, scorers=[response_length]) without predict_fn

   Why: Leaving out predict_fn scores the stored traces without calling the app

   You should see: A response_length metric for each trace, with no new LLM calls from the app