CertSafari
    Databricks Certified Generative AI Engineer Associate· Lessons

    Domain 6 · Lesson 50/56

    Inference logs for diagnosing and scoring a deployed RAG app

    Use inference logging to assess deployed RAG application performance

    16 min read
    1.2% of exam
    6 sources
    Published 22 Oct 2026
    Docs as of 17 Oct 2026

    What you will be able to do

    • Pick the inference table columns that answer a given RAG question, such as which payload columns to join against SME ground truth
    • Query logged traffic for slow requests, failing queries and suspect retrievals, and name what the log cannot show
    • Explain why the unified trace table is easier to score than raw inference table payloads
    • Schedule production scorers on a sample of live traces and choose sample rates that balance cost against coverage

    Key concept

    Logged traffic versus scored traffic — An inference log records what happened on each request: the payloads, the latency and the status. It does not record whether the answer was any good. To measure RAG quality after launch, run scorers over a sample of the logged traces and attach their verdicts to those traces.

    1.Which logged columns answer RAG questions

    Once a RAG application is live, its inference log is the only record of the questions real users asked and the answers they got. An inference table writes every request and response from a serving endpoint to a Delta table in Unity Catalog. How to enable it, its full schema and its gaps are covered in Inference tables: enabling, schema and gaps. This lesson is about reading that table as evidence of how the RAG application is performing.

    Two columns hold almost all of the RAG content: request and response. Each one stores the raw JSON payload as a string. The user's question is in request. The generated answer is in response, and so is any retrieved context the application returns with it. Every other column describes the call (when it happened, how long it took, who made it, whether it succeeded) and holds none of the conversation.

    Unity Gateway inference table columns, mapped to the RAG question each one answers
    ColumnWhat it holdsRAG question it answers
    requestThe raw JSON request payloadWhat did the user actually ask?
    responseThe raw JSON response payloadWhat answer was returned, and with what retrieved context?
    latency_ms / time_to_first_byte_msTotal latency and time to first byteWhich requests were slow, and did users wait before seeing any output?
    status_codeHTTP status code of the responseWhich queries failed outright?
    logging_error_codesError codes if logging failedWhich rows are missing their payload (MAX_REQUEST_SIZE_EXCEEDED, MAX_RESPONSE_SIZE_EXCEEDED)?
    sampling_fractionSampling fraction if down-sampling was usedAm I looking at all traffic or a sample?
    request_id / invocation_idUnique identifiers for the request and each inference callWhich calls belong together? (identifiers only, no content)

    The payload columns are also what you use to build datasets. Suppose an engineer wants a fine-tuning and evaluation set that pairs real production questions with answers written by SMEs. The production side of that join has to come from request and response, because those are the only columns that contain the text. Identifier columns such as databricks_request_id and client_request_id (their names in the legacy schema) only track calls. status_code and requester record the outcome and the caller. Timing columns record when the call happened and how long it ran. None of these hold a question or an answer that a label could be matched against.

    Agent endpoints log more than the payload pair. When inference tables are enabled for a deployed agent, they also capture the MLflow trace. The trace breaks one request into its steps, so a retrieval call and a generation call show up separately instead of as a single JSON blob. In the agent's payload request-log table, the request column is reduced to the last user query in the conversation, which makes it easier to read than the raw body.

    Checkpoint 1 of 8· Match them up

    An engineer is using a RAG endpoint's inference table to check how it performs in production. Match each column to the question it can answer.

    Tap a term, then the definition that fits it.

    Checkpoint 2 of 8· Exam question

    An ML engineer wants to build a new fine-tuning and evaluation dataset by combining real production traffic with SME-provided correct answers. They plan to join AI Gateway inference table records against a table of human-labeled ground truth. Which pair of inference table columns provides the raw content needed for this join?

    Sources12

    2.Diagnosing slow, failing and badly grounded requests

    You query the table directly from Databricks SQL or a notebook. Start broad, then narrow down by the symptom you're chasing. Each RAG problem leaves a different mark in the log.

    The starting point for every diagnosis: query the inference table directlysql
    SELECT * FROM <catalog>.<schema>.<payload_table>

    For slow requests, sort or filter on latency_ms, which is the total latency of the call, and on time_to_first_byte_ms. A large gap between the two on a streaming chat endpoint means users saw the first token quickly and then waited for the rest of the answer. The legacy schema measures time differently: its execution_duration_ms counts only the time the model spent generating predictions and excludes network overhead. A latency number from one schema therefore can't be compared directly with a number from the other.

    For empty or irrelevant retrievals, the evidence lives inside response. If the chain returns the retrieved chunks with its answer, you can parse that JSON in SQL and look for requests where the context list is empty or where the sources don't match the topic of the question in request. These rows nearly always have status 200. An empty retrieval doesn't fail the call. It produces a confident answer with nothing behind it.

    For failing queries, status_code is the obvious column, but it has a blind spot. Rate-limit rejections, permission errors and server errors may be missing from the table altogether, so a quiet error count is not proof that nothing failed. Compare it with the endpoint health metrics, which report error rate as an infrastructure measure.

    Several other gaps matter specifically for RAG:

    • A single latency number per row can't tell you whether retrieval or generation was slow. You need the step-level trace to split them.
    • Large payloads disappear. Requests or responses over 10 MiB aren't logged, which can happen when a RAG response carries many retrieved chunks. Filter on logging_error_codes to find these rows before you conclude a request was empty.
    • Fresh traffic lags. Rows can take up to an hour to appear after the endpoint receives its first request, so a dashboard built on a brand-new table will look empty at first.
    • Delivery is at-least-once, so the same request can appear twice. Deduplicate before you compute rates.
    • No row carries a quality verdict. Nothing in the schema says whether an answer was grounded in its context or relevant to the question.

    Checkpoint 3 of 8· Check yourself

    Your RAG chain returns its retrieved chunks along with each answer. In the inference table, some rows for long, chunk-heavy answers have no response payload. A teammate concludes that retrieval came back empty for those questions. What should you check before you accept that conclusion?

    Sources123

    3.From per-endpoint payloads to the unified trace table

    Raw payload logs are awkward to score at scale. Each inference table covers one endpoint and uses a Databricks-specific layout. Turning its rows into something MLflow scorers can read means extracting traces by hand. A RAG agent that calls an embedding endpoint, a retriever and an LLM ends up with its evidence spread across several tables.

    The unified trace table fixes this by writing every request and response from all Unity Gateway services to one Unity Catalog table in OpenTelemetry span format. A metastore admin sets it up once, and no per-endpoint configuration follows. It is in Beta and is the recommended approach for new deployments.

    Inference tables compared with the unified trace table, for scoring RAG traffic
    AspectInference tablesUnified trace table
    ScopeOne table per model serving endpointAll Unity Gateway model and MCP services in one table
    SetupEnabled per endpointOne-time setup by a metastore admin
    SchemaDatabricks-specific, requires post-processingOpenTelemetry spans
    Agentic workflowsFragmented across per-endpoint tablesAll hops land in one table
    MLflow compatibilityRequires manual trace extractionMLflow-compatible OTel schema that MLflow tools read directly
    Default accessEndpoint ownerMetastore admins only, until access is granted

    Spans also make step-level diagnosis possible. Filter on trace_id to replay every step of a failed run in order. That shows whether a bad answer started at the retrieval span or the generation span, which a single inference table row can't do. To find failures across an endpoint, filter on service_name and status.code. The table is clustered by time, so include the time column in the WHERE clause.

    Finding every error on one endpoint in the last day, using span statussql
    -- All errors on a specific endpoint in the last 24 hours
    SELECT trace_id, name, status, attributes
    FROM <catalog>.<schema>.<table_name>
    WHERE service_name = '<catalog>.<schema>.<model>'
      AND status.code = 'STATUS_CODE_ERROR'
      AND time >= current_timestamp() - INTERVAL 1 DAY;

    Two cautions apply. Access is locked down by default: only the metastore admin who created the table can query it, so the RAG team needs an explicit grant before they can run the query above. Trace delivery is also best-effort. The table supplements the Databricks audit logs and does not replace them as the compliance record.

    Checkpoint 4 of 8· Check yourself

    A RAG agent calls an embedding endpoint, a retriever and an LLM. The team wants to see whether bad answers start at the retrieval step or the generation step. They also want to score the traffic with MLflow tools without extracting traces by hand. Which setup fits?

    Sources4

    4.Scoring live traffic with production monitoring

    Logs tell you a request happened. Scorers tell you whether it was good. Production monitoring schedules MLflow 3 scorers against an MLflow experiment. The monitoring service evaluates a configurable sample of incoming traces and attaches each result to its trace as feedback. Groundedness and relevance, the questions the inference table schema can't answer, become numbers you can track over time.

    This is the right tool for tracking quality after launch, and re-running the pre-launch test set is not. A fixed offline set of question-answer pairs measures the same questions every week, whether you run it weekly or add more pairs to it. Production monitoring scores the questions real users are asking today, and it can use the same scorers you built during development, so production and development numbers are comparable. The feature is in Beta.

    Three prerequisites have to be in place before you register anything:

    • The application must log traces with MLflow Tracing to an MLflow experiment. Traces logged to individual runs instead of the experiment are a common reason scorers never execute.
    • If traces are stored in Unity Catalog, the monitoring job queries them through a SQL warehouse. Set its ID with set_databricks_monitoring_sql_warehouse_id(), which stores it as an experiment tag. The MLFLOW_TRACING_SQL_WAREHOUSE_ID environment variable only applies to your own process, and the monitoring job never sees it.
    • Custom scorers must be self-contained @scorer functions registered from a Databricks notebook, with their imports inside the function body. Class-based Scorer subclasses can't be serialized for remote execution.

    Scoring starts when you start a scorer. Each scorer follows the same .register() then .start() pattern, whether it is a built-in judge, a custom judge or a code-based scorer. ScorerSamplingConfig controls how many traces get scored, and its filter_string controls which ones.

    Registering a judge and scoring only successful tracespython
    from mlflow.genai.scorers import Safety, ScorerSamplingConfig
    
    # Only evaluate traces that completed successfully
    safety_judge = Safety().register(name="safety")
    safety_judge = safety_judge.start(
        sampling_config=ScorerSamplingConfig(
            sample_rate=1.0,
            filter_string="attributes.status = 'OK'"
        ),
    )

    Set each sample rate by what the scorer costs to run and what a miss would cost you. Critical checks such as safety and security run at sample_rate=1.0. Expensive LLM judges like groundedness run at 0.05 to 0.2, which is enough to show a trend without paying to judge every request. Moderate rates of 0.3 to 0.5 suit iterative improvement. Turning a judge off, or relying only on status codes, cuts the cost by removing the quality signal entirely. Increasing payload capture doesn't touch scoring cost at all.

    Two operational limits matter here. An experiment can have at most 20 scorers associated with it for continuous monitoring. The first results take 15 to 20 minutes to appear after scheduling, and at low sample rates they arrive more slowly. Results show up in the experiment's Traces tab and in the monitoring dashboards. The monitoring job runs as the user who registered the first scorer, so that user's permissions determine which traces it can read.

    Checkpoint 5 of 8· Fill the gap

    This judge should score only the traces that completed successfully. Which parameter of ScorerSamplingConfig completes the sample?

    from mlflow.genai.scorers import Safety, ScorerSamplingConfig
    
    # Only evaluate traces that completed successfully
    safety_judge = Safety().register(name="safety")
    safety_judge = safety_judge.start(
        sampling_config=ScorerSamplingConfig(
            sample_rate=1.0,
             ? ="attributes.status = 'OK'"
        ),
    )

    Checkpoint 6 of 8· Check yourself

    The traces for a RAG agent are stored in Unity Catalog. An engineer sets MLFLOW_TRACING_SQL_WAREHOUSE_ID in their notebook and then registers and starts a groundedness judge. The monitoring job still can't query the traces. What fixes it?

    Checkpoint 7 of 8· Exam question

    A production monitoring job for a deployed RAG agent runs an expensive LLM-as-judge scorer that checks groundedness on every incoming trace, and the team notices monitoring costs rising sharply as traffic grows. Which change addresses the cost concern while still giving ongoing visibility into groundedness quality?

    Checkpoint 8 of 8· Exam question

    Before launch, a team ran a curated offline evaluation of their RAG agent using a fixed test set of question-answer pairs in MLflow. After deployment, they want to keep track of answer quality as real users interact with the agent day to day. Which approach fits this post-launch need?

    Sources56

    Sources

    (6)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “The raw JSON request payload.”
      ↩︎ Which logged columns answer RAG questions
      “Error responses: Logs may not be populated for requests that return 401, 403, 429, or 500 errors.”
      ↩︎ Diagnosing slow, failing and badly grounded requests
      “Requests and responses larger than 10 MiB aren't logged.”
      ↩︎ Diagnosing slow, failing and badly grounded requests
      “Rows can take up to an hour to appear in the table after the endpoint receives its first request.”
      ↩︎ Diagnosing slow, failing and badly grounded requests
      “Inference tables deliver logs through Zerobus and share its latency, at-least-once delivery, and retention behavior.”
      ↩︎ Diagnosing slow, failing and badly grounded requests
      “Error responses: Logs may not be populated for requests that return 401, 403, 429, or 500 errors.”
      ↩︎ Exam trap 2
      “request | STRING | The raw JSON request payload.”
      ↩︎ Checkpoint
      “Requests and responses larger than 10 MiB aren't logged. The logging_error_codes column indicates when this occurs with MAX_REQUEST_SIZE_EXCEEDED or MAX_RESPONSE_SIZE_EXCEEDED.”
      ↩︎ Checkpoint
    2. 2.
      “By joining inference tables with ground truth labels, you can create a training corpus”
      ↩︎ Which logged columns answer RAG questions
      “these inference tables store payload and request details as well as MLflow Trace logs”
      ↩︎ Which logged columns answer RAG questions
      “The last user query from the user's conversation.”
      ↩︎ Which logged columns answer RAG questions
      “This does not include overhead network latencies and only represents the time it took for the model to generate predictions.”
      ↩︎ Diagnosing slow, failing and badly grounded requests
      “By joining inference tables with ground truth labels, you can create a training corpus”
      ↩︎ Exam trap 1
    3. 3.
      “Provides insights into infrastructure metrics like latency, request rate, error rate, CPU usage, and memory usage.”
      ↩︎ Diagnosing slow, failing and badly grounded requests
    4. 4.
      “Inference tables remain available but are designed for single model-service endpoint, request/response monitoring only.”
      ↩︎ From per-endpoint payloads to the unified trace table
      “Spans use MLflow-compatible OTel schema that MLflow tools can read directly”
      ↩︎ From per-endpoint payloads to the unified trace table
      “Filter by service_name and status.code to find all errors on a specific endpoint.”
      ↩︎ From per-endpoint payloads to the unified trace table
      “By default, only the metastore admin who creates the trace table can read it.”
      ↩︎ From per-endpoint payloads to the unified trace table
      “Trace delivery is best-effort (see Limitations), so the unified trace table complements rather than replaces the Databricks audit logs.”
      ↩︎ From per-endpoint payloads to the unified trace table
      “Filter by trace_id to replay every step of a failed agent run.”
      ↩︎ Checkpoint
    5. 5.
      “Use the same scorers in development and production to ensure consistent evaluation.”
      ↩︎ Scoring live traffic with production monitoring
      “Instrumented production application: Your agent must log traces using MLflow Tracing.”
      ↩︎ Scoring live traffic with production monitoring
      “For critical scorers such as safety and security checks, use sample_rate=1.0.”
      ↩︎ Scoring live traffic with production monitoring
      “At any given time, at most 20 scorers can be associated with an experiment for continuous quality monitoring.”
      ↩︎ Scoring live traffic with production monitoring
      “Only @scorer decorator-based scorers can be registered.”
      ↩︎ Scoring live traffic with production monitoring
      “Configurable sampling rates so you can control the tradeoff between coverage and computational cost.”
      ↩︎ Exam trap 3
      “Setting the MLFLOW_TRACING_SQL_WAREHOUSE_ID environment variable in your notebook or application is not a substitute.”
      ↩︎ Exam trap 4
      “For expensive scorers, such as complex LLM judges, use lower sample rates (0.05-0.2).”
      ↩︎ Prediction
      “The monitoring job runs separately and reads the warehouse ID from the experiment tag.”
      ↩︎ Checkpoint
    6. 6.
      “It closes the loop — the same scorers you built during evaluation now watch production.”
      ↩︎ Scoring live traffic with production monitoring
      “Production monitoring continuously runs MLflow scorers on a sample of your live production traces and attaches the results as feedback”
      ↩︎ Key concept

    Spotted a mistake, or was something unclear? Tell us.