Lakehouse Monitoring Profile Types, Baselines and Drift Tests

1.The parts of a model monitoring pipeline

A production monitoring pipeline answers four separate questions. Logging: what did the model receive and return? Drift detection: has the data changed? Model performance: are the predictions still good? Model health: is the serving infrastructure working? On Databricks, the first three come from Lakehouse Monitoring. The current documentation calls this feature *data profiling*. It computes summary statistics over time on a Unity Catalog table. For ML, you point it at an inference table, which is the request log of model inputs and predictions.

The fourth question has a separate answer. Serving endpoints expose their own health metrics, such as latency, request rate, error rate, CPU usage and memory usage, which describe the infrastructure, not the data. Keep the two apart. A monitor on an inference table cannot tell you a replica is out of memory, and an endpoint's CPU chart cannot tell you a feature has drifted.

Matching the table type to the need. Start from the table you have and the question you want answered. If the table is a model's request log (timestamp, inputs, prediction and optionally a ground-truth label) and you need to compare model performance and data quality across time windows, use an inference table with an inference profile. If the table is a time series with a timestamp column and you need data quality metrics per time window, use a time series profile. Use a snapshot profile for every other table, where you want metrics over all the data on each refresh. The reason is that only the inference profile adds model quality metrics, only the windowed types give you consecutive drift between time windows, and a snapshot is limited to 4TB.

Applying the statistical tests. Once a monitor is running, the drift metrics table is where you test for change. For a numerical column, read ks_test, which returns a statistic and a p-value, along with wasserstein_distance and population_stability_index. For a categorical column, read chi_squared_test, which also returns a statistic and a p-value, along with tv_distance, l_infinity_distance and js_distance. To evaluate the significance of an observed change, filter the rows to the column, slice and drift type you care about. A small p-value, or a distance or PSI that is large compared with your alert threshold, marks a change worth investigating. The section on drift tests covers how to read each metric.

Sources123

2.Choosing snapshot, time series or inference

When you create a monitor, you attach it to one table, the *primary table*, and choose one of three analysis types. Choose the type from the table's structure, not from what you hope to learn from it.

Two limits often come up in exam scenarios. First, a snapshot profile reprocesses the whole table on every refresh, so it stops at 4TB. Second, time series and inference profiles only compute metrics over the last 30 days. A question asking about trends over several quarters is testing whether you know that limit.

Sources1

3.Drift against a baseline or against the previous window

The drift metrics table holds two kinds of comparison, marked in its drift_type column. CONSECUTIVE compares a window with the previous window, and is only produced if a previous window exists at the chosen granularity. BASELINE compares a window with an optional baseline table. Each kind catches different changes. Consecutive drift picks up a sudden change from one week to the next. Baseline drift picks up a slow slide away from the expected data that no single step would reveal.

A baseline table should show what *acceptable* data looks like, and it should match the primary table's schema, except for the timestamp column. What counts as acceptable depends on the profile type. For a snapshot, use a past snapshot of good quality. For a time series, use a period with normal values. For an inference profile, use the model's training or validation data, so that you are alerted when live inputs move away from what the model learned from. That table also needs the same model_id_col as the primary table, so the data is aggregated the same way.

Detecting drift in practice. To compare the current data distribution with a known baseline, supply the baseline table when you create the monitor, then query the drift metrics table for rows where drift_type is BASELINE. To compare the current distribution with successive time windows, query the rows where drift_type is CONSECUTIVE. The window column holds the current window and window_cmp holds the previous window it was compared against. All the differences are calculated as the current window minus the comparison window. In both cases you read the same distribution metrics, so the choice is only about the reference you want: a fixed picture of expected data, or the window just before.

Sources13

4.Statistical tests in the drift metrics table

Each drift row is for one column in one window, slice and drift type. Simple deltas such as count_delta, avg_delta and percent_null_delta are calculated as the current window minus the comparison window. Distribution drift uses different tests depending on the column type. If a test does not apply to a column, its cell is null.

To judge how significant a change is, you need to know what each metric gives you. The Kolmogorov-Smirnov and chi-squared tests report both a statistic and a p-value. The docs do not set a p-value cutoff, so your alert defines the threshold. Population stability index is the one metric with documented bands: under 0.1 means no significant change, under 0.2 moderate, and 0.2 or above significant. Remember too that the distribution tests and the integrity deltas measure different things. A column whose null rate jumps shows up in percent_null_delta even if its distribution test stays quiet.

Sources3

5.Model performance trends from an inference table

An inference profile adds model quality metrics to the profile metrics table. For classification these include accuracy_score, precision, recall, f1_score and confusion_matrix. For regression they include mean_squared_error, root_mean_squared_error and r2_score. These metrics apply to the whole table, so their column_name is :table. They only appear if both a label column and a prediction column are set. Without ground truth, you can still watch input and prediction drift, but you cannot see accuracy.

Every metric is also computed for each model id, so comparing version A with version B is just a filter on that column. The docs show this query on an InferenceLog profile:

SELECT
  window.start, column_name, count, num_nulls, distinct_count, frequent_items
FROM census_monitor_db.adult_census_profile_metrics
WHERE model_id = 1    -- Constrain to version 1
  AND slice_key IS NULL    -- look at aggregate metrics over the whole data
  AND column_name = "income_predicted"
ORDER BY window.start

Sources1

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The KS test can flag drift in a categorical column such as region.Why is that wrong?

   KS is calculated only for numeric columns. Categorical columns use chi-squared, TV, L-infinity and JS distance, and the numeric-only cells are null for them.

   Covered in Statistical tests in the drift metrics table

2. 2.Without a baseline table, a monitor produces no drift metrics at all.Why is that wrong?

   Consecutive drift compares adjacent windows and needs no baseline. Only baseline drift depends on a baseline table.

   Covered in Drift against a baseline or against the previous window

3. 3.An inference profile always reports accuracy because it knows the prediction column.Why is that wrong?

   Model accuracy metrics appear only if both a label column and a prediction column are configured.

   Covered in Model performance trends from an inference table