Collect SME Feedback with the MLflow Review App

1.Why domain experts review traces

Automated judges and metrics only know what you tell them. A domain expert knows what a correct answer looks like for your business. Databricks puts it plainly: one of the most effective ways to improve your agent is to have domain experts review and label existing traces. The tool built for this is the MLflow Review App. It gives you a structured way to collect expert feedback on real interactions with your application.

The documentation lists three uses for the Review App. Experts help you understand what high-quality, correct responses look like for specific queries. They collect input to align LLM judges with your business requirements. And their labels help you create evaluation datasets from production traces. All three come back in the rest of this lesson.

Choosing traces is the first real decision. Focus on cases where a human adds something an automated judge cannot. That includes traces with ambiguous or borderline quality, edge cases not covered by automated judges, and examples where automated metrics disagree with expected quality. Add representative samples of different user interaction patterns so the labels aren't all edge cases. In the MLflow UI you can filter traces by status, tags, or time range. You can also select them through the SDK.

One note on direction: Databricks says that for new human-review workflows it recommends review queues (Beta), which route items to reviewers one at a time. The sources say little more about review queues, so this lesson covers labeling sessions and the Chat UI, which the docs describe in full.

Sources12

2.Two ways to use the Review App: Chat UI or labeling existing traces

The Review App works in two ways, and they fit different situations.

The Chat UI lets experts talk to a live version of your agent and rate each answer as they go. Databricks calls it a way to vibe check your app. It is good for testing multi-turn conversations and for checking an update in a safe environment before production. You get it by calling get_review_app() and connecting your endpoint with review_app.add_agent(...). Then you share the URL with the experts. The Chat UI has fixed feedback questions, so you don't define a labeling schema. Every interaction and its feedback is automatically saved as a trace in MLflow.

Labeling existing traces is the more systematic way. You choose traces that are already logged, and experts answer questions that you define. It also covers deployments the Chat UI can't reach.

Both modes need two things from every reviewer. They must be provisioned in your Databricks account, and they must have the Workspace access entitlement for the workspace that hosts the experiment. The docs warn that account provisioning alone is not enough. The other permissions depend on the mode, as the table shows.

Sources31

3.Labeling sessions: packaging traces for expert review

To label existing traces, you organise them into a labeling session. A labeling session is a special type of MLflow run that holds the set of traces you want experts to review. Because it is an MLflow run, it shows up in the Evaluations tab of the MLflow UI, and you can query it with mlflow.search_runs().

When you create a session you set its name, the assigned users (the experts), an optional agent (this links the session to the Chat UI), the labeling schemas (the questions experts answer), and whether to allow multi-turn chat. Through the API, at least one schema is required.

import mlflow.genai.labeling as labeling
import mlflow.genai.label_schemas as schemas

# Create a simple labeling session with built-in schemas
session = labeling.create_labeling_session(
    name="customer_service_review_jan_2024",
    assigned_users=["alice@company.com", "bob@company.com"],
    label_schemas=[schemas.EXPECTED_FACTS]  # Required: at least one schema needed
)

A new session is empty. After you create it, you must add traces for experts to review. In the UI, open the experiment's Traces view, tick the traces you want, and from the Actions drop-down menu select Add to labeling session. In code, use LabelingSession.add_traces. To bring in reviewers, open the session and click Share, then enter each reviewer's email address. Reviewers are notified and given access to the Review App. Their answers appear under Assessments when you open a request in the session.

One detail matters for automation. Session names might not be unique, so store and look up sessions by their MLflow run ID (session.mlflow_run_id), not by name.

Sources4

4.Labeling schemas: deciding what experts are asked

A labeling schema sets the questions an expert answers when labeling existing traces. It controls the question text, the input method (for example a drop-down or a text box), validation rules, and optional instructions. Each schema produces one assessment on the trace. Schemas are scoped to an experiment, so a name must be unique within that experiment. They are not used in the Chat UI.

Every schema has one of two types, matching the key concept of this lesson. A feedback schema collects subjective assessments such as ratings, preferences, or opinions. An expectation schema collects objective ground truth such as correct answers or expected behavior.

# Expectation schema for ground truth
facts_schema = schemas.create_label_schema(
    name="required_facts",
    type="expectation",
    title="What facts must be included in a correct response?",
    input=InputTextList(max_count=5, max_length_each=200),
    instruction="List key facts that any correct response must contain."
)

MLflow also ships predefined schema names that the built-in judges read. If you collect expert ground truth under one of these names, the matching built-in judge can use it without extra wiring. To change a schema later, call create_label_schema again with overwrite=True.

Sources5

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The Review App Chat UI can connect to any deployed agent, including one running on Databricks Apps.Why is that wrong?

   The Chat UI needs an agent deployed to a Model Serving endpoint. For Databricks Apps deployments, you label existing traces instead.

   Covered in Two ways to use the Review App: Chat UI or labeling existing traces

2. 2.You must define a custom labeling schema before experts can give feedback in the Chat UI.Why is that wrong?

   Schemas apply only when labeling existing traces. The Chat UI uses fixed feedback questions.

   Covered in Labeling schemas: deciding what experts are asked

3. 3.A labeling session's name uniquely identifies it, so pipelines can look sessions up by name.Why is that wrong?

   Session names can repeat. Use the MLflow run ID to store and reference a session.

   Covered in Labeling sessions: packaging traces for expert review