Model Monitoring in Snowflake: Drift, AUC, Precision, Recall and RMSE

1.Why deployed models decay, and what a monitor watches

A model that scored well at training time is not guaranteed to keep scoring well. Snowflake's ML Observability documentation names the usual causes: the inputs change, the assumptions baked in at training go stale, and upstream pipelines break. Traffic also changes over time. This gradual loss of quality is what the exam guide calls model decay. Deciding whether to retrain starts with measuring the deployed model against two references: the data it was trained on, and the outcomes that actually happened.

Snowflake does this measuring with a model version monitor. The monitor works on stored inference data. When you run a registered model, its predictions come back as a DataFrame, and teams usually write them to a Snowflake table even when inference runs outside Snowflake. Each row in the monitoring logs holds an ID, a timestamp, the features, the prediction, and (once known) the ground-truth value. Keeping features and predictions together over time is what makes both questions in the exam guide answerable: has the input data moved away from training, and are the predictions themselves shifting?

A monitor is tied to one model version. It refreshes its logs by querying the source data, and it aggregates them over an aggregation window, currently at least one day. An optional baseline table holds the reference data that drift is measured against. Monitors support regression, binary classification and multi-class classification models, and only single-output models.

CREATE [ OR REPLACE ] MODEL MONITOR [ IF NOT EXISTS ] <monitor_name> WITH
    MODEL = <model_name>
    VERSION = '<version_name>'
    FUNCTION = '<function_name>'
    SOURCE = <source_name>
    WAREHOUSE = <warehouse_name>
    REFRESH_INTERVAL = '<num> { seconds | minutes | hours | days }'
    AGGREGATION_WINDOW = '<num> days'
    TIMESTAMP_COLUMN = <timestamp_name>
    [ BASELINE = <baseline_name> ]
    [ ID_COLUMNS = <id_column_name_array> ]
    [ PREDICTION_CLASS_COLUMNS = <prediction_class_column_name_array> ]
    [ PREDICTION_SCORE_COLUMNS = <prediction_column-name_array> ]
    [ ACTUAL_CLASS_COLUMNS = <actual_class_column_name_array> ]
    [ ACTUAL_SCORE_COLUMNS = <actual_column_name_array> ]
    [ SEGMENT_COLUMNS = <segment_column_name_array> ]
    [ CUSTOM_METRIC_COLUMNS = <custom_metric_column_name_array> ]
    [ COMMENT = '<string_literal>' ]

The column rules matter later. Timestamp columns must be TIMESTAMP_NTZ, and prediction and actual columns must be NUMBER. Each column can appear only once across the monitor's parameters, so the ID column cannot also serve as the prediction column. Invalid values, such as nulls, NaNs, infinities or probability scores outside 0–1, can make the monitor fail and suspend. A monitor reports three families of metrics: drift, performance, and statistical counts such as nulls.

Sources1

2.Data drift: comparing production data with the baseline

The exam guide's first drift question is: do the data making predictions look like the training data? In Snowflake, you answer it with the baseline. The baseline table holds a snapshot of data similar to the source, and that snapshot is stored inside the monitor object. Drift metrics compare the distribution of a column in production with the same column in the baseline, one time bucket at a time. Without a baseline, MODEL_MONITOR_DRIFT_METRIC returns an error.

The column you compare does not have to be an input. For a model version monitor, column_name can be a feature column, a prediction column or an actual column. Drift on a feature answers "are the inputs different from training?" Drift on the prediction column answers "has the model's output distribution moved?" That is how the sources address the exam guide's second question about whether the deployed model behaves as it did. The sources do not describe a row-by-row check that identical inputs still produce identical predictions. What they offer is distribution drift on the prediction column, plus the stored log of IDs, features and predictions you can query yourself.

Four drift metric names are valid: JENSEN_SHANNON, DIFFERENCE_OF_MEANS, WASSERSTEIN and POPULATION_STABILITY_INDEX. The reference gives no formulas or alert thresholds for them, so don't rely on memorised cut-offs here. (One example on the observability page passes 'PSI', but the function reference lists POPULATION_STABILITY_INDEX as the valid value.) Requesting a numerical drift metric for a non-numeric feature is listed as an error. The optional arguments are granularity (default 1 DAY), start time (default 60 days ago) and end time (default now).

SELECT * FROM TABLE( ? (
'MY_MONITOR', 'JENSEN_SHANNON', 'MODEL_PREDICTION', '1 DAY', DATEADD('DAY', -30, CURRENT_DATE()), CURRENT_DATE())
)

Drift works alongside a simpler signal. MODEL_MONITOR_STAT_METRIC returns COUNT, COUNT_NULL, MIN, MAX, AVG and SUM for a column. The count metrics work on any column type, while the other four work only on numeric columns. A sudden jump in nulls or a collapse in row count usually points to a broken pipeline, not real-world drift, and the two call for different fixes.

Sources234

3.Accuracy, precision and recall: what they measure and what the monitor needs

Drift tells you the data has moved. It does not tell you the model is now wrong. To measure that, you need ground truth. For classifiers, Snowflake's documentation builds the metrics from four counts: true positives (TP, correct predictions of positive instances), true negatives (TN), false positives (FP, incorrect predictions of positive instances) and false negatives (FN).

- Precision = TP / (TP + FP): of everything predicted positive, how much really was positive. - Recall (sensitivity) = TP / (TP + FN): of everything really positive, how much the model caught. - F1 is the harmonic mean of precision and recall. - Accuracy = (TP + TN) / all predictions. The documentation warns this metric can be misleading in unbalanced cases.

In a model monitor, these metrics come from MODEL_MONITOR_PERFORMANCE_METRIC, and the names you can request depend on the model type. Binary classifiers accept CLASSIFICATION_ACCURACY, PRECISION, RECALL, F1_SCORE and ROC_AUC. Multi-class classifiers accept CLASSIFICATION_ACCURACY plus MACRO_AVERAGE_PRECISION, MACRO_AVERAGE_RECALL, MICRO_AVERAGE_PRECISION and MICRO_AVERAGE_RECALL. The sources name the macro and micro variants without defining how each one averages, so learn the names here. The documentation also notes that for binary classification you can use the micro-average metrics much as you would use classification accuracy in multi-class classification.

Every accuracy-family metric needs actual values. Actual columns are optional when you create the monitor, but if you leave them out these metrics are not computed. Leaving the actual column empty also causes errors.

Sources56

4.Area under the ROC curve: a score-based metric

Here is why AUC is different. A classifier turns a probability into a class by comparing it with a threshold: a sample belongs to a class if its predicted probability exceeds the threshold. Precision, recall and accuracy are each measured at one threshold. Snowflake's classification documentation describes threshold metrics that sweep the threshold from 0 to 1. At each threshold it computes the true positive rate (TPR, the same as recall) and the false positive rate (FPR, the share of actual negatives wrongly predicted positive). These pairs can be used to plot ROC and PR curves. The area under the ROC curve therefore summarises performance across all thresholds, which is why it needs the raw score. A hard class label has already lost the threshold information.

In practice, if your inference table stores only a 0/1 label, ROC_AUC cannot be computed. When you design the source table, keep the probability in a PREDICTION_SCORE_COLUMNS column as well as the class. Remember that probability scores outside 0–1 count as invalid values that can suspend the monitor.

AUC also differs on confidence intervals. The performance function's CI_VALUE column returns a Wilson interval for CLASSIFICATION_ACCURACY, PRECISION, RECALL, F1_SCORE and the micro-average metrics. ROC_AUC is not in that list, so its CI_VALUE is NULL.

Sources56

5.RMSE for regression, then reading and acting on the results

Regression models get a separate set of performance metrics: RMSE, MAE, MAPE and MSE. RMSE, MAE and MAPE all need prediction_score and actual_score, because a regression output is a number, not a class. The sources do not give the RMSE formula. They expand the name (Root Mean Square Error), list its required columns, and say that CI_VALUE for RMSE, MSE and MAE is a Wald interval, where the classification metrics use Wilson. The query pattern matches the other metrics, here with daily buckets over the last 30 days:

SELECT * FROM TABLE(MODEL_MONITOR_PERFORMANCE_METRIC(
'MY_MONITOR', 'RMSE', '1 DAY', DATEADD('DAY', -30, CURRENT_DATE()), CURRENT_DATE())
)

Each result row has EVENT_TIMESTAMP, METRIC_VALUE, COUNT_USED and COUNT_UNUSED. Check the counts: a metric computed on few records is weak evidence either way.

An overall average can hide a segment that is failing. If the monitor was created with SEGMENT_COLUMNS, you can pass a JSON extra_args value to get any drift, performance or stat metric for one segment. Segment columns must be string categorical columns, with at most 5 per monitor (a hard limit) and fewer than 25 unique values each (a recommendation). Segment values are case sensitive, special characters are not supported in segment queries, and each call accepts only one column:value pair.

'{"SEGMENTS": [{"column": "<segment_column_name>", "value": "<segment_value>"}]}'

You can view results in two places. The Snowsight ML Monitoring dashboard (AI & ML » Models, then a model's Monitors list) plots metrics over time and can compare two version monitors. The metric functions return rows you can feed into Streamlit or another monitoring tool. You can also set up alerts and notifications on monitoring metrics. The sources here point to a separate Alerts and Notifications page and do not describe its mechanics.

A monitor can also stop reporting on its own. After five consecutive refresh failures related to the source tables, it suspends refreshes. DESCRIBE MODEL MONITOR shows aggregation_status (with SUSPENDED values) and aggregation_last_error (the SQL error). After you fix the cause, ALTER MODEL MONITOR … RESUME restarts refreshes. A flat dashboard may mean a suspended monitor, not a healthy model.

Rising drift or falling precision, recall, AUC or RMSE quality is the evidence for retraining. How you schedule and promote a retrained version is covered in the neighbouring lessons, not here.

Sources61

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A monitor created without a BASELINE can be given one later with ALTER MODEL MONITOR so drift starts working.Why is that wrong?

   Drift needs baseline data, and a baseline cannot be added afterwards. You must drop the monitor and create it again with BASELINE set.

   Covered in Data drift: comparing production data with the baseline

2. 2.ROC_AUC can be computed from the same prediction_class column used for precision and recall.Why is that wrong?

   ROC_AUC reads the predicted score, not the class, so the monitor needs a prediction_score column alongside actual_class.

   Covered in Area under the ROC curve: a score-based metric

3. 3.High accuracy on an imbalanced dataset shows the model is performing well.Why is that wrong?

   Accuracy counts true negatives, so a dominant negative class can inflate it while recall on the rare class stays poor.

   Covered in Accuracy, precision and recall: what they measure and what the monitor needs

4. 4.One segment query can filter on REGION and CHANNEL together by passing two entries in SEGMENTS.Why is that wrong?

   Each call accepts only one segment column:value pair, so querying two segments takes two separate calls.

   Covered in RMSE for regression, then reading and acting on the results