Validating Models in Snowflake: Confusion Matrix, Thresholds, ROC and Regression Metrics

1.Where validation numbers come from

Validation answers one question: how well does the model do on data it has not seen? Snowflake's documentation states the goal directly: metrics measure how accurately a model predicts new data. Before you read any metric, you need to know how Snowflake produced it.

For a classification model built with CREATE SNOWFLAKE.ML.CLASSIFICATION, Snowflake does not score the model on its own training rows. It takes a random sample from the full dataset and trains a separate model without those rows. It then runs inference on the held-back rows and compares the predicted classes with the actual ones. The model you deploy is still trained on all the data. The evaluation copy exists only to produce honest numbers.

Evaluation must be turned on. Each evaluation method (SHOW_EVALUATION_METRICS, SHOW_GLOBAL_EVALUATION_METRICS, SHOW_THRESHOLD_METRICS, SHOW_CONFUSION_MATRIX) returns results only for models where evaluation was enabled when the model was created. Each result row is labelled with a dataset_type, which is currently EVAL. That label tells you the numbers come from the held-out sample.

Forecasting models follow the same idea, adapted for time series. By default the forecasting function evaluates every model it trains with cross-validation. It trains extra models on subsets of the training data and predicts the withheld periods.

Sources123

2.Reading the confusion matrix

The confusion matrix is the most basic output of that comparison. SHOW_CONFUSION_MATRIX returns one row for each combination of actual class and predicted class. The count column holds the number of evaluation rows that fall into that combination. The method takes no arguments. Below is the documented output for a binary model:

+--------------+--------------+-----------------+-------+------+
| DATASET_TYPE | ACTUAL_CLASS | PREDICTED_CLASS | COUNT | LOGS |
|--------------+--------------+-----------------+-------+------|
| EVAL         | false        | false           |    37 | NULL |
| EVAL         | false        | true            |     1 | NULL |
| EVAL         | true         | false           |     0 | NULL |
| EVAL         | true         | true            |    22 | NULL |
+--------------+--------------+-----------------+-------+------+

Take true as the positive class and read the rows as pairs. Actual true predicted true (22) is a true positive. Actual false predicted false (37) is a true negative. Actual false predicted true (1) is a false positive. Actual true predicted false (0) is a false negative. That makes 60 evaluation rows, 59 of them on the diagonal. The documentation states the goal: maximize the instances on the diagonal and minimize those off it.

To see this as a chart in Snowsight, run the call, click Chart, and set Chart Type to Heatgrid. Under Data, set Cell values to NONE, Rows to PREDICTED_CLASS and Columns to ACTUAL_CLASS. Note the orientation: predicted classes run down the rows and actual classes across the columns. If you assume the opposite layout, you will swap false positives and false negatives when you read the heatgrid.

Sources1

3.Thresholds, ROC curves and expected payout

A confusion matrix is a snapshot at one decision threshold. The classifier actually outputs a probability for each class. A row is assigned to a class only if that probability exceeds the threshold. Each threshold gives a different set of true and false positives and negatives. SHOW_THRESHOLD_METRICS exposes all of these. For each class and each threshold, it returns raw counts plus the metrics derived from them. The documentation says this output can be used to plot ROC and PR curves or to tune the threshold. For a multi-class model, each class is treated one-vs-rest: every instance that does not belong to the class is counted as negative.

ROC. The tpr and fpr columns at each threshold are the points of the ROC curve. Precision and recall at each threshold give the PR curve. AUC is one of the metrics Snowflake averages across classes. For binary problems, the classification function is trained with an area-under-the-curve loss. Remember that AUC summarises all thresholds at once, while the model is deployed at a single threshold. When positives are rare, a small false-positive *rate* can still mean many false alerts compared with true ones. Check precision at your chosen threshold, not just AUC.

Expected payout. The Snowflake documentation does not define a payout or profit metric. It provides what you need to compute one yourself: the tp, fp, tn and fn counts at every threshold. The business supplies the value of each outcome. As an illustration with assumed values (not from the documentation): suppose a caught positive earns 100, a false alarm costs 20, and a missed positive costs 100. For the 60-row matrix in the previous section, that gives 22×100 − 1×20 − 0×100 = 2,180, or about 36 per evaluated row. Repeat this for every threshold row and choose the threshold with the best total. That is threshold tuning measured in money instead of accuracy.

Sources14

4.Regression problems: error metrics and residuals

Regression models predict a number, not a class, so there is no confusion matrix. Instead, error metrics summarise the gap between predicted and actual values. Snowflake uses the same family of metrics in several places. A model monitor attached to a regression model accepts RMSE, MAE, MAPE and MSE as performance metrics. Each requires both a prediction_score column and an actual_score column. Without actual values, there is no error to measure. For MSE, RMSE and MAE, the monitor also reports a 95% Wald confidence interval in CI_VALUE.

A forecasting model's SHOW_EVALUATION_METRICS returns these point metrics from cross-validation: MAE, MAPE, MDA (mean directional accuracy), MSE and SMAPE. It adds interval metrics: COVERAGE_INTERVAL, the share of actual values that fall inside the prediction interval, and the Winkler score. You can also call it with new out-of-sample data to see how the model handles periods it has never seen.

+--------+--------------------------+--------------+--------------------+------+
| SERIES | ERROR_METRIC             | METRIC_VALUE | STANDARD_DEVIATION | LOGS |
+--------+--------------------------+--------------+--------------------+------+
| NULL   | "MAE"                    |         2.49 |                NaN | NULL |
| NULL   | "MAPE"                   |        0.084 |                NaN | NULL |
| NULL   | "MDA"                    |         0.99 |                NaN | NULL |
| NULL   | "MSE"                    |        8.088 |                NaN | NULL |
| NULL   | "SMAPE"                  |        0.077 |                NaN | NULL |
| NULL   | "WINKLER_ALPHA=0.05"     |       12.101 |                NaN | NULL |
| NULL   | "COVERAGE_INTERVAL=0.95" |            1 |                NaN | NULL |
+--------+--------------------------+--------------+--------------------+------+

The STANDARD_DEVIATION column is empty here for a documented reason. With the default n_splits of 1, only one validation set exists, so there is no spread to report. Increase n_splits if you want to see how stable each metric is across folds.

With a Snowpark ML regressor, you validate the same way you would in scikit-learn. Split the data, fit on the training portion, predict the held-out portion, and score it. The Snowflake developer guide splits with random_split([0.85, 0.15], seed=42), trains an XGBRegressor, and then computes test MSE and R² on the predictions:

from sklearn.metrics import mean_squared_error, r2_score

predictions = regressor.predict(test_df)
predictions_pd = predictions.to_pandas()

mse = mean_squared_error(predictions_pd["ETA_MINUTES"], predictions_pd["predicted_eta"])
r2 = r2_score(predictions_pd["ETA_MINUTES"], predictions_pd["predicted_eta"])

print(f"Test MSE: {mse:.2f}")
print(f"Test R²: {r2:.2f}")

Sources562

5.Model metrics: which output answers which question

The classification evaluation APIs split the work across four methods. Choosing the right one is often what an exam question is really testing. Per-class precision, recall and F1 come from SHOW_EVALUATION_METRICS. One overall number per metric comes from SHOW_GLOBAL_EVALUATION_METRICS, which averages the per-class values. Its average_type column is currently MACRO, so every class counts equally regardless of its size. Logistic loss is calculated once for the whole model. F1 is the harmonic mean of precision and recall. The documentation recommends it as the balancing metric when class distributions are uneven.

After deployment, the same metric families continue in model monitoring. For a binary classification model, MODEL_MONITOR_PERFORMANCE_METRIC accepts ROC_AUC, CLASSIFICATION_ACCURACY, PRECISION, RECALL and F1_SCORE. For a multi-class model, it accepts accuracy plus macro- and micro-averaged precision and recall. ROC_AUC needs prediction_score and actual_class, because AUC is built from scores. The class-based metrics need prediction_class and actual_class. Do not count support as a quality score. It describes how often a class appears in the dataset, not how the model performs.

CALL <model_name>!SHOW_EVALUATION_METRICS();
CALL <model_name>! ? ();
CALL <model_name>!SHOW_THRESHOLD_METRICS();
CALL <model_name>!SHOW_CONFUSION_MATRIX();

Sources715

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.You can call SHOW_CONFUSION_MATRIX or SHOW_THRESHOLD_METRICS on any classification model after training.Why is that wrong?

   These methods return results only for models where evaluation was enabled when the model was created. The metrics come from a separately trained evaluation model scored on withheld rows.

   Covered in Where validation numbers come from

2. 2.High accuracy, or a high ROC-AUC, on a rare-positive problem means the model's alerts are mostly correct.Why is that wrong?

   Accuracy is dominated by the majority class and can mislead on unbalanced data. Check precision and the false-positive count at the deployed threshold, which SHOW_THRESHOLD_METRICS reports per threshold.

   Covered in Thresholds, ROC curves and expected payout

3. 3.A class with high support is a class the model predicts well.Why is that wrong?

   Support only counts how often the class actually occurs (true positives plus false negatives). It says nothing about prediction quality.

   Covered in Model metrics: which output answers which question

4. 4.A model monitor can report RMSE or MAE from predictions alone.Why is that wrong?

   Regression error metrics compare predictions with actual values. The monitor needs both prediction_score and actual_score, and actual columns are required for accuracy metrics.

   Covered in Regression problems: error metrics and residuals