What you will be able to do
- Pick the inference table columns that answer a given RAG question, such as which payload columns to join against SME ground truth
- Query logged traffic for slow requests, failing queries and suspect retrievals, and name what the log cannot show
- Explain why the unified trace table is easier to score than raw inference table payloads
- Schedule production scorers on a sample of live traces and choose sample rates that balance cost against coverage
Key concept
Logged traffic versus scored traffic — An inference log records what happened on each request: the payloads, the latency and the status. It does not record whether the answer was any good. To measure RAG quality after launch, run scorers over a sample of the logged traces and attach their verdicts to those traces.
1.Which logged columns answer RAG questions
Once a RAG application is live, its inference log is the only record of the questions real users asked and the answers they got. An inference table writes every request and response from a serving endpoint to a Delta table in Unity Catalog. How to enable it, its full schema and its gaps are covered in Inference tables: enabling, schema and gaps. This lesson is about reading that table as evidence of how the RAG application is performing.
Two columns hold almost all of the RAG content: request and response. Each one stores the raw JSON payload as a string. The user's question is in request. The generated answer is in response, and so is any retrieved context the application returns with it. Every other column describes the call (when it happened, how long it took, who made it, whether it succeeded) and holds none of the conversation.
| Column | What it holds | RAG question it answers |
|---|---|---|
| request | The raw JSON request payload | What did the user actually ask? |
| response | The raw JSON response payload | What answer was returned, and with what retrieved context? |
| latency_ms / time_to_first_byte_ms | Total latency and time to first byte | Which requests were slow, and did users wait before seeing any output? |
| status_code | HTTP status code of the response | Which queries failed outright? |
| logging_error_codes | Error codes if logging failed | Which rows are missing their payload (MAX_REQUEST_SIZE_EXCEEDED, MAX_RESPONSE_SIZE_EXCEEDED)? |
| sampling_fraction | Sampling fraction if down-sampling was used | Am I looking at all traffic or a sample? |
| request_id / invocation_id | Unique identifiers for the request and each inference call | Which calls belong together? (identifiers only, no content) |
The payload columns are also what you use to build datasets. Suppose an engineer wants a fine-tuning and evaluation set that pairs real production questions with answers written by SMEs. The production side of that join has to come from request and response, because those are the only columns that contain the text. Identifier columns such as databricks_request_id and client_request_id (their names in the legacy schema) only track calls. status_code and requester record the outcome and the caller. Timing columns record when the call happened and how long it ran. None of these hold a question or an answer that a label could be matched against.
Agent endpoints log more than the payload pair. When inference tables are enabled for a deployed agent, they also capture the MLflow trace. The trace breaks one request into its steps, so a retrieval call and a generation call show up separately instead of as a single JSON blob. In the agent's payload request-log table, the request column is reduced to the last user query in the conversation, which makes it easier to read than the raw body.
Checkpoint 1 of 8· Match them up
An engineer is using a RAG endpoint's inference table to check how it performs in production. Match each column to the question it can answer.
Tap a term, then the definition that fits it.
Only the request and response columns hold the conversation, stored as raw JSON. Every other column describes the call itself. Identifiers like request_id only track calls, so a join with SME ground truth has to match on the payload text.
“request | STRING | The raw JSON request payload.”Source: docs.databricks.com
Checkpoint 2 of 8· Exam question
An ML engineer wants to build a new fine-tuning and evaluation dataset by combining real production traffic with SME-provided correct answers. They plan to join AI Gateway inference table records against a table of human-labeled ground truth. Which pair of inference table columns provides the raw content needed for this join?
Correct answer: A — request and response, which hold the raw JSON payloads sent to and returned from the endpoint
- A. The
requestandresponsecolumns store the actual JSON payloads exchanged with the endpoint, which is the content needed to pair a user's real question and the model's real answer with a ground-truth label for building a training or eval corpus. - B. The identifier columns are useful for tracing and deduplication but contain no question or answer content, so they alone cannot supply the text needed to build a labeled dataset.
- C. HTTP status and requester identity describe the outcome and caller of a request, not the conversational content, so they don't provide the material needed to pair with SME labels.
- D. Timestamp and latency fields describe when and how fast a call ran, which supports performance analysis but does not contain the question or answer text required for a labeled dataset.
2.Diagnosing slow, failing and badly grounded requests
You query the table directly from Databricks SQL or a notebook. Start broad, then narrow down by the symptom you're chasing. Each RAG problem leaves a different mark in the log.
SELECT * FROM <catalog>.<schema>.<payload_table>For slow requests, sort or filter on latency_ms, which is the total latency of the call, and on time_to_first_byte_ms. A large gap between the two on a streaming chat endpoint means users saw the first token quickly and then waited for the rest of the answer. The legacy schema measures time differently: its execution_duration_ms counts only the time the model spent generating predictions and excludes network overhead. A latency number from one schema therefore can't be compared directly with a number from the other.
For empty or irrelevant retrievals, the evidence lives inside response. If the chain returns the retrieved chunks with its answer, you can parse that JSON in SQL and look for requests where the context list is empty or where the sources don't match the topic of the question in request. These rows nearly always have status 200. An empty retrieval doesn't fail the call. It produces a confident answer with nothing behind it.
For failing queries, status_code is the obvious column, but it has a blind spot. Rate-limit rejections, permission errors and server errors may be missing from the table altogether, so a quiet error count is not proof that nothing failed. Compare it with the endpoint health metrics, which report error rate as an infrastructure measure.
Several other gaps matter specifically for RAG:
- A single latency number per row can't tell you whether retrieval or generation was slow. You need the step-level trace to split them.
- Large payloads disappear. Requests or responses over 10 MiB aren't logged, which can happen when a RAG response carries many retrieved chunks. Filter on
logging_error_codesto find these rows before you conclude a request was empty. - Fresh traffic lags. Rows can take up to an hour to appear after the endpoint receives its first request, so a dashboard built on a brand-new table will look empty at first.
- Delivery is at-least-once, so the same request can appear twice. Deduplicate before you compute rates.
- No row carries a quality verdict. Nothing in the schema says whether an answer was grounded in its context or relevant to the question.
Checkpoint 3 of 8· Check yourself
Your RAG chain returns its retrieved chunks along with each answer. In the inference table, some rows for long, chunk-heavy answers have no response payload. A teammate concludes that retrieval came back empty for those questions. What should you check before you accept that conclusion?
A missing payload can mean the payload was too large to log, which is likely when a RAG response carries many retrieved chunks. logging_error_codes marks those rows, so check it before you diagnose an empty retrieval.
“Requests and responses larger than 10 MiB aren't logged. The logging_error_codes column indicates when this occurs with MAX_REQUEST_SIZE_EXCEEDED or MAX_RESPONSE_SIZE_EXCEEDED.”Source: docs.databricks.com
3.From per-endpoint payloads to the unified trace table
Raw payload logs are awkward to score at scale. Each inference table covers one endpoint and uses a Databricks-specific layout. Turning its rows into something MLflow scorers can read means extracting traces by hand. A RAG agent that calls an embedding endpoint, a retriever and an LLM ends up with its evidence spread across several tables.
The unified trace table fixes this by writing every request and response from all Unity Gateway services to one Unity Catalog table in OpenTelemetry span format. A metastore admin sets it up once, and no per-endpoint configuration follows. It is in Beta and is the recommended approach for new deployments.
| Aspect | Inference tables | Unified trace table |
|---|---|---|
| Scope | One table per model serving endpoint | All Unity Gateway model and MCP services in one table |
| Setup | Enabled per endpoint | One-time setup by a metastore admin |
| Schema | Databricks-specific, requires post-processing | OpenTelemetry spans |
| Agentic workflows | Fragmented across per-endpoint tables | All hops land in one table |
| MLflow compatibility | Requires manual trace extraction | MLflow-compatible OTel schema that MLflow tools read directly |
| Default access | Endpoint owner | Metastore admins only, until access is granted |
Spans also make step-level diagnosis possible. Filter on trace_id to replay every step of a failed run in order. That shows whether a bad answer started at the retrieval span or the generation span, which a single inference table row can't do. To find failures across an endpoint, filter on service_name and status.code. The table is clustered by time, so include the time column in the WHERE clause.
-- All errors on a specific endpoint in the last 24 hours
SELECT trace_id, name, status, attributes
FROM <catalog>.<schema>.<table_name>
WHERE service_name = '<catalog>.<schema>.<model>'
AND status.code = 'STATUS_CODE_ERROR'
AND time >= current_timestamp() - INTERVAL 1 DAY;Two cautions apply. Access is locked down by default: only the metastore admin who created the table can query it, so the RAG team needs an explicit grant before they can run the query above. Trace delivery is also best-effort. The table supplements the Databricks audit logs and does not replace them as the compliance record.
Checkpoint 4 of 8· Check yourself
A RAG agent calls an embedding endpoint, a retriever and an LLM. The team wants to see whether bad answers start at the retrieval step or the generation step. They also want to score the traffic with MLflow tools without extracting traces by hand. Which setup fits?
A metastore admin sets up the unified trace table once. It then puts every hop into one table as OpenTelemetry spans that MLflow tools can read directly, and filtering on trace_id replays each step in order. Only the metastore admin can query it until access is granted. Per-endpoint inference tables split the run across several tables and need traces extracted by hand.
“Filter by trace_id to replay every step of a failed agent run.”Source: docs.databricks.com
Sources4
4.Scoring live traffic with production monitoring
Logs tell you a request happened. Scorers tell you whether it was good. Production monitoring schedules MLflow 3 scorers against an MLflow experiment. The monitoring service evaluates a configurable sample of incoming traces and attaches each result to its trace as feedback. Groundedness and relevance, the questions the inference table schema can't answer, become numbers you can track over time.
This is the right tool for tracking quality after launch, and re-running the pre-launch test set is not. A fixed offline set of question-answer pairs measures the same questions every week, whether you run it weekly or add more pairs to it. Production monitoring scores the questions real users are asking today, and it can use the same scorers you built during development, so production and development numbers are comparable. The feature is in Beta.
Three prerequisites have to be in place before you register anything:
- The application must log traces with MLflow Tracing to an MLflow experiment. Traces logged to individual runs instead of the experiment are a common reason scorers never execute.
- If traces are stored in Unity Catalog, the monitoring job queries them through a SQL warehouse. Set its ID with
set_databricks_monitoring_sql_warehouse_id(), which stores it as an experiment tag. TheMLFLOW_TRACING_SQL_WAREHOUSE_IDenvironment variable only applies to your own process, and the monitoring job never sees it. - Custom scorers must be self-contained
@scorerfunctions registered from a Databricks notebook, with their imports inside the function body. Class-basedScorersubclasses can't be serialized for remote execution.
Scoring starts when you start a scorer. Each scorer follows the same .register() then .start() pattern, whether it is a built-in judge, a custom judge or a code-based scorer. ScorerSamplingConfig controls how many traces get scored, and its filter_string controls which ones.
from mlflow.genai.scorers import Safety, ScorerSamplingConfig
# Only evaluate traces that completed successfully
safety_judge = Safety().register(name="safety")
safety_judge = safety_judge.start(
sampling_config=ScorerSamplingConfig(
sample_rate=1.0,
filter_string="attributes.status = 'OK'"
),
)Set each sample rate by what the scorer costs to run and what a miss would cost you. Critical checks such as safety and security run at sample_rate=1.0. Expensive LLM judges like groundedness run at 0.05 to 0.2, which is enough to show a trend without paying to judge every request. Moderate rates of 0.3 to 0.5 suit iterative improvement. Turning a judge off, or relying only on status codes, cuts the cost by removing the quality signal entirely. Increasing payload capture doesn't touch scoring cost at all.
Two operational limits matter here. An experiment can have at most 20 scorers associated with it for continuous monitoring. The first results take 15 to 20 minutes to appear after scheduling, and at low sample rates they arrive more slowly. Results show up in the experiment's Traces tab and in the monitoring dashboards. The monitoring job runs as the user who registered the first scorer, so that user's permissions determine which traces it can read.
Checkpoint 5 of 8· Fill the gap
This judge should score only the traces that completed successfully. Which parameter of ScorerSamplingConfig completes the sample?
from mlflow.genai.scorers import Safety, ScorerSamplingConfig
# Only evaluate traces that completed successfully
safety_judge = Safety().register(name="safety")
safety_judge = safety_judge.start(
sampling_config=ScorerSamplingConfig(
sample_rate=1.0,
? ="attributes.status = 'OK'"
),
)In ScorerSamplingConfig, sample_rate sets how many traces get scored and filter_string sets which ones. In this sample, filter_string limits the judge to traces whose status is OK.
Source: docs.databricks.comCheckpoint 6 of 8· Check yourself
The traces for a RAG agent are stored in Unity Catalog. An engineer sets MLFLOW_TRACING_SQL_WAREHOUSE_ID in their notebook and then registers and starts a groundedness judge. The monitoring job still can't query the traces. What fixes it?
The environment variable applies only to the notebook's own process. The monitoring job runs separately and reads the warehouse ID from an experiment tag, which set_databricks_monitoring_sql_warehouse_id() writes.
“The monitoring job runs separately and reads the warehouse ID from the experiment tag.”Source: docs.databricks.com
Checkpoint 7 of 8· Exam question
A production monitoring job for a deployed RAG agent runs an expensive LLM-as-judge scorer that checks groundedness on every incoming trace, and the team notices monitoring costs rising sharply as traffic grows. Which change addresses the cost concern while still giving ongoing visibility into groundedness quality?
Correct answer: A — Lower the scorer's sampling rate so groundedness is evaluated on a representative subset of traces instead of all traffic
- A. Reducing the sampling rate lets a scheduled scorer evaluate a representative fraction of live traces rather than every single one, which is the documented way to balance evaluation coverage against the cost of running an expensive LLM judge continuously.
- B. HTTP status codes only indicate whether a call completed successfully, not whether the response was grounded in retrieved context, so dropping the scorer entirely would eliminate the quality signal the team needs.
- C. Restricting groundedness checks to offline evaluation before deployment removes the ability to detect quality drift on real production traffic, which defeats the purpose of production monitoring.
- D. The payload size limit controls whether an individual request or response body is captured, not how many traces a scheduled scorer processes, so raising it would not reduce scoring cost or address the sampling tradeoff.
Checkpoint 8 of 8· Exam question
Before launch, a team ran a curated offline evaluation of their RAG agent using a fixed test set of question-answer pairs in MLflow. After deployment, they want to keep track of answer quality as real users interact with the agent day to day. Which approach fits this post-launch need?
Correct answer: A — Register scheduled scorers against the deployed agent so a sampled portion of live inference traffic is continuously evaluated for quality
- A. Scheduled scorers applied to sampled production traces are the monitoring-phase mechanism for tracking a live agent, since they continuously assess real user interactions rather than a static, pre-defined test set.
- B. Re-running a fixed offline test set repeatedly checks the agent against the same curated questions, not against how real users are actually interacting with it, so it would not surface issues specific to live traffic patterns.
- C. HTTP status codes only show whether a call succeeded technically; they say nothing about whether the generated answer was accurate, relevant, or grounded, so they cannot substitute for quality scoring.
- D. Growing the offline test set improves pre-deployment evaluation coverage but still only reflects curated scenarios, not the evolving distribution of real production questions the agent receives after launch.
Sources
(6)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“The raw JSON request payload.”
↩︎ Which logged columns answer RAG questions“Error responses: Logs may not be populated for requests that return 401, 403, 429, or 500 errors.”
↩︎ Diagnosing slow, failing and badly grounded requests“Requests and responses larger than 10 MiB aren't logged.”
↩︎ Diagnosing slow, failing and badly grounded requests“Rows can take up to an hour to appear in the table after the endpoint receives its first request.”
↩︎ Diagnosing slow, failing and badly grounded requests“Inference tables deliver logs through Zerobus and share its latency, at-least-once delivery, and retention behavior.”
↩︎ Diagnosing slow, failing and badly grounded requests“Error responses: Logs may not be populated for requests that return 401, 403, 429, or 500 errors.”
↩︎ Exam trap 2“request | STRING | The raw JSON request payload.”
↩︎ Checkpoint“Requests and responses larger than 10 MiB aren't logged. The logging_error_codes column indicates when this occurs with MAX_REQUEST_SIZE_EXCEEDED or MAX_RESPONSE_SIZE_EXCEEDED.”
↩︎ Checkpoint - 2.
“By joining inference tables with ground truth labels, you can create a training corpus”
↩︎ Which logged columns answer RAG questions“these inference tables store payload and request details as well as MLflow Trace logs”
↩︎ Which logged columns answer RAG questions“The last user query from the user's conversation.”
↩︎ Which logged columns answer RAG questions“This does not include overhead network latencies and only represents the time it took for the model to generate predictions.”
↩︎ Diagnosing slow, failing and badly grounded requests“By joining inference tables with ground truth labels, you can create a training corpus”
↩︎ Exam trap 1 - 3.https://docs.databricks.com/aws/en/machine-learning/model-serving/monitor-diagnose-endpointsOfficial docs
“Provides insights into infrastructure metrics like latency, request rate, error rate, CPU usage, and memory usage.”
↩︎ Diagnosing slow, failing and badly grounded requests - 4.
“Inference tables remain available but are designed for single model-service endpoint, request/response monitoring only.”
↩︎ From per-endpoint payloads to the unified trace table“Spans use MLflow-compatible OTel schema that MLflow tools can read directly”
↩︎ From per-endpoint payloads to the unified trace table“Filter by service_name and status.code to find all errors on a specific endpoint.”
↩︎ From per-endpoint payloads to the unified trace table“By default, only the metastore admin who creates the trace table can read it.”
↩︎ From per-endpoint payloads to the unified trace table“Trace delivery is best-effort (see Limitations), so the unified trace table complements rather than replaces the Databricks audit logs.”
↩︎ From per-endpoint payloads to the unified trace table“Filter by trace_id to replay every step of a failed agent run.”
↩︎ Checkpoint - 5.
“Use the same scorers in development and production to ensure consistent evaluation.”
↩︎ Scoring live traffic with production monitoring“Instrumented production application: Your agent must log traces using MLflow Tracing.”
↩︎ Scoring live traffic with production monitoring“For critical scorers such as safety and security checks, use sample_rate=1.0.”
↩︎ Scoring live traffic with production monitoring“At any given time, at most 20 scorers can be associated with an experiment for continuous quality monitoring.”
↩︎ Scoring live traffic with production monitoring“Only @scorer decorator-based scorers can be registered.”
↩︎ Scoring live traffic with production monitoring“Configurable sampling rates so you can control the tradeoff between coverage and computational cost.”
↩︎ Exam trap 3“Setting the MLFLOW_TRACING_SQL_WAREHOUSE_ID environment variable in your notebook or application is not a substitute.”
↩︎ Exam trap 4“For expensive scorers, such as complex LLM judges, use lower sample rates (0.05-0.2).”
↩︎ Prediction“The monitoring job runs separately and reads the warehouse ID from the experiment tag.”
↩︎ Checkpoint - 6.
“It closes the loop — the same scorers you built during evaluation now watch production.”
↩︎ Scoring live traffic with production monitoring“Production monitoring continuously runs MLflow scorers on a sample of your live production traces and attaches the results as feedback”
↩︎ Key concept