Built-in LLM Judges That Require Ground Truth

9 min read

1.79% of exam

Published 3 Oct 2026

Docs as of 30 Sep 2026

What you will be able to do

Explain what ground truth means in an MLflow evaluation dataset and where it is stored
Name the built-in single-turn judges that require ground truth: Correctness, RetrievalSufficiency and ToolCallCorrectness
Tell apart look-alike judge pairs where one needs ground truth and the other does not
Recognise that every built-in multi-turn conversation judge is reference-free

Key concept

Ground truth (expectations) — Ground truth is the human-supplied expected result for an evaluation row, such as the expected answer, the expected facts or the expected tool calls. In MLflow it is stored in the optional expectations field. A judge 'requires ground truth' when it can only score a response by comparing it against that field.

1.Where ground truth lives: the expectations field

Before you can say which judges need ground truth, you need to know what ground truth looks like to MLflow. An evaluation dataset row has a required inputs dictionary, which holds what you send to the app. It can also have an optional expectations dictionary, which holds what a human says the right result is. When you score pre-computed outputs or existing traces with mlflow.genai.evaluate(), expectations is still optional. The harness calls it "Optional ground truth for scorers."

Because the field is optional, you can run an evaluation without any labels at all. Judges that read only inputs and outputs will still produce scores. Judges that compare against an expected answer will not have the data they need. That difference is what the rest of this lesson is about.

Built-in judges look for specific keys inside expectations. The dataset reference lists reserved keys, and each one belongs to a particular consumer:

Reserved keys inside expectations and the judge or scorer that reads them

Key	Used by	Holds
expected_facts	Correctness judge	List of facts that should appear
expected_response	Correctness judge	Exact or similar expected output
guidelines	Guidelines judge	Natural language rules to follow
expected_retrieved_context	document_recall scorer	Documents that should be retrieved

Checkpoint 1 of 6· Check yourself

A dataset row has inputs and outputs but no expectations. Which statement is accurate?

Sources1 2

2.The built-in single-turn judges: which ones say "Yes"

Built-in LLM judges are predefined scorers that run on Databricks-hosted LLMs. Each one documents its arguments and whether it requires ground truth. The arguments column is the quickest tell. A judge that lists expectations among its arguments reads labelled data, and a judge that lists only inputs, outputs does not.

Built-in single-turn LLM judges and whether they require ground truth

Judge	Arguments	Requires ground truth
RelevanceToQuery	inputs, outputs	No
RetrievalRelevance	inputs, outputs	No
Safety	inputs, outputs	No
RetrievalGroundedness	inputs, outputs	No
Correctness	inputs, outputs, expectations	Yes
RetrievalSufficiency	inputs, outputs, expectations	Yes
Guidelines	inputs, outputs	No
ExpectationsGuidelines	inputs, outputs, expectations	No (but needs guidelines in expectations)
ToolCallCorrectness	inputs, outputs, expectations	Yes
ToolCallEfficiency	inputs, outputs	No

Exactly three built-in single-turn judges answer "Yes". Correctness asks whether the response is correct compared with the provided ground truth. RetrievalSufficiency asks whether the retrieved context could support a response that includes the ground-truth facts. ToolCallCorrectness asks whether the tool calls and their arguments were correct for the query. Each one judges against a known right answer: the right facts, the right context or the right tool calls. That is why each needs a human to supply that answer first.

Checkpoint 2 of 6· Match them up

Match each built-in judge to its ground-truth requirement

Tap a term, then the definition that fits it.

Checkpoint 3 of 6· Exam question

A support-quality team has a labeled dataset of chat transcripts, each paired with a known correct answer written by subject-matter experts. They want an MLflow built-in judge that scores whether the agent's response matches this known correct answer for each transaction. Which judge should they use, and why does it require ground truth?

Sources3

3.Look-alike pairs and the ExpectationsGuidelines edge case

Most mistakes come from judges whose names sound alike but sit on different sides of the line.

RetrievalGroundedness vs RetrievalSufficiency. Groundedness asks whether the response is grounded in the context, or whether the agent is hallucinating. It only needs the response and the context the app retrieved, so it is reference-free. Sufficiency asks whether that context contained enough to produce the ground-truth facts, so it needs expectations.

ToolCallEfficiency vs ToolCallCorrectness. Efficiency asks whether the tool calls avoided redundancy, which can be judged from the calls alone. Correctness asks whether the calls and their arguments were right, which needs expected tool calls.

Guidelines vs ExpectationsGuidelines. Guidelines applies one global set of natural-language rules to every row and takes no expectations. ExpectationsGuidelines applies per-row guidelines that domain experts labelled in an evaluation dataset. Its table entry reads "No (but needs guidelines in expectations)". It does not compare against a correct answer, but it still reads the expectations field, where the per-row guidelines are stored. That limits where it can run: the Guidelines page marks Guidelines as working in both offline evaluation and production monitoring, and ExpectationsGuidelines as for offline evaluation only.

Checkpoint 4 of 6· Fill the gap

Per-row guidelines for ExpectationsGuidelines go under which key?

from mlflow.genai.scorers import ExpectationsGuidelines
import mlflow

# Dataset with per-row guidelines
data = [
    {
        "inputs": {"question": "What is the capital of France?"},
        "outputs": "The capital of France is Paris.",
        " ? ": {
            "guidelines": ["The response must be factual and concise"]
        }
    },

Checkpoint 5 of 6· Exam question

A RAG engineering team has a benchmark set where each question has expert-written expected_facts. They want to confirm the retriever pulled documents that actually contain enough information to support producing those expected facts, not merely documents that are topically relevant. Which built-in judge fits this need and requires ground truth?

Sources3 4

4.Multi-turn conversation judges

For conversational agents, MLflow also provides judges that score a whole conversation instead of a single turn. They take a session argument; ConversationalGuidelines also takes guidelines. The judges are ConversationCompleteness, UserFrustration, KnowledgeRetention, ConversationalGuidelines, ConversationalRoleAdherence, ConversationalSafety and ConversationalToolCallEfficiency. All seven are marked "No" for ground truth. Each one assesses patterns visible in the conversation history itself, such as whether every question was addressed, whether the user got frustrated and whether the agent stayed in role. The docs say to use them both for evaluation during development and for monitoring in production.

Checkpoint 6 of 6· Check yourself

A team wants to check whether their assistant stays in its assigned role across a conversation, using unlabelled production traffic. What is accurate about ConversationalRoleAdherence?