What you will be able to do
- Compute precision, recall and F1 from confusion counts and explain why accuracy misleads on imbalanced labels
- Choose between macro, weighted and micro averaging when F1 is extended to several classes
- Read a ROC curve and ROC AUC from thresholded scores, and say when precision-recall AUC separates models that ROC AUC does not
- Explain what log loss measures, why it punishes confident wrong predictions, and which direction is better for each metric
Key concept
What a classification metric consumes — Each classification metric reads one of three things: hard labels after a threshold (accuracy, precision, recall, F1), the ordering of scores across thresholds (ROC AUC, precision-recall AUC), or the probability given to the true class (log loss). What a metric reads decides what it can detect.
1.Confusion counts and why accuracy misleads on rare positives
A binary classifier turns a probability into a label with a threshold, and every metric in this section is computed from the four counts that come out. A row that is truly positive and predicted positive is a true positive (tp). A truly negative row predicted positive is a false positive (fp). A positive the model missed is a false negative (fn), and a correctly rejected negative is a true negative (tn). The confusion matrix just lays those four counts out in a grid.
# Predict and evaluate on the held-out test set
y_pred_proba = final_model.predict(dtest)
y_pred = (y_pred_proba > 0.5).astype(int)
accuracy = accuracy_score(y_test, y_pred)
test_logloss = log_loss(y_test, y_pred_proba)Accuracy is (tp + tn) divided by every row, so the majority class dominates it. In the fraud set the 9,800 true negatives swamp the 200 rows that matter. A 98% score therefore tells the fraud team nothing until they see how many of the 200 frauds were caught. Scikit-learn's balanced accuracy exists to expose this: it falls back to chance level when plain accuracy beats chance only because the classifier takes advantage of an imbalanced test set. Rebalancing the training data is one fix, covered in Class imbalance: undersampling, oversampling/SMOTE, class weights. Whatever you do to the training data, report F1, precision and recall or PR-AUC on imbalanced labels, never accuracy alone.
Checkpoint 1 of 10· Fill the gap
This Databricks tutorial computes accuracy from labels thresholded at 0.5 and log loss on the same test set. Which argument completes the log_loss call?
# Predict and evaluate on the held-out test set
y_pred_proba = final_model.predict(dtest)
y_pred = (y_pred_proba > 0.5).astype(int)
accuracy = accuracy_score(y_test, y_pred)
test_logloss = log_loss(y_test, ? )Log loss scores the probability the model gave to the true class, so it needs the raw probabilities in y_pred_proba. Accuracy reads the labels after the 0.5 threshold, and passing those thresholded labels to log loss would throw away the confidence it is meant to grade.
Source: docs.databricks.com2.Precision, recall and F1 from the same counts
Precision and recall each ignore the true negatives, and that is why they work on rare positives. Precision is tp / (tp + fp): of the rows the model flagged, the fraction that really were positive. Recall is tp / (tp + fn): of the real positives, the fraction the model found. Recall also goes by true positive rate and sensitivity. Put intuitively, precision is the ability not to label a negative sample as positive, and recall is the ability to find all the positive samples.
F1 is the harmonic mean of the two: F1 = 2 × precision × recall / (precision + recall). A harmonic mean is pulled towards the smaller value, so F1 rises only when both are good. Go back to the fraud set. The flag-nothing model has recall 0, which makes its F1 0 even though its accuracy is 98%. A model that flags 250 transactions, 150 of them real fraud, has precision 150/250 = 0.6 and recall 150/200 = 0.75. Its F1 is 2 × 0.6 × 0.75 / 1.35 ≈ 0.67. In scikit-learn's own small example, precision 1.0 and recall 0.5 give F1 ≈ 0.67, not the arithmetic mean of 0.75. F1 is the general F-measure with β = 1. Other values of β weight recall more or less heavily, and F1's best value is 1, its worst 0.
| Model | tp | fp | fn | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|
| Flags nothing | 0 | 0 | 200 | 0.98 | undefined (scikit-learn returns 0 by default) | 0.0 | 0.0 |
| Flags 250, 150 correct | 150 | 100 | 50 | 0.985 | 0.6 | 0.75 | ≈0.67 |
These metrics depend on which class you call positive. Scikit-learn's binary metrics score only the positive label, assumed to be 1 unless you set pos_label. Databricks AutoML's classify takes a pos_label argument for binary problems for the same reason, since it is needed to compute precision and recall. If the label is reversed, you are measuring how well the model finds legitimate transactions.
Fewer rows get flagged, so the false positives usually drop and precision rises. Some real frauds now score below 0.9, so false negatives grow and recall falls. F1 moves only if the gain in one outweighs the loss in the other, which is why it is reported at a stated threshold.
Checkpoint 2 of 10· Check yourself
In a payments table, fraud is stored as 0 and legitimate as 1. A data scientist calls f1_score(y_test, y_pred) with no other arguments and reports the 0.97 it returns as the model's fraud-detection F1. What does that 0.97 actually measure?
Binary metrics such as F1 score only the positive label, which defaults to 1. Here 1 means legitimate, so the score shows how well the model finds legitimate transactions. To score fraud, set pos_label=0 or recode the labels.
“by default only the positive label is evaluated, assuming by default that the positive class is labelled 1”Source: scikit-learn.org
Checkpoint 3 of 10· Exam question
A data scientist at a payments company trains a classifier to flag fraudulent transactions in a dataset where only 2% of transactions are fraudulent. The model reports 98% accuracy, but the fraud team is skeptical that this reflects real detection ability. Which metric should the team report instead to better capture how well the model balances catching fraud against raising false alarms?
Correct answer: A — F1 score is the right choice because it combines precision and recall into a single harmonic-mean value that only rises when the model both catches real fraud and avoids flagging too many legitimate transactions.
- A. The F1 score is the harmonic mean of precision and recall, so it stays low unless the model both catches a meaningful share of true fraud cases and keeps false alarms in check. That combination makes it far more informative than a raw correctness percentage on a dataset this skewed.
- B. Accuracy does not account for class imbalance at all; a model that predicts every transaction as legitimate would still score 98% accuracy on this dataset while catching zero fraud. It is misleading here precisely because it rewards ignoring the minority class.
- C. Root mean squared error is a regression metric for continuous numeric targets, not the standard tool for evaluating a fraud/not-fraud classification decision. It does not summarize precision or recall tradeoffs the fraud team cares about.
- D. R-squared measures explained variance for continuous outcomes and is not a standard way to evaluate a binary classifier's ability to separate fraud from legitimate transactions. It gives no direct read on false positives or false negatives.
3.Averaging F1 across several classes
F1 is a binary metric. With several classes it is computed once per class, treating that class as positive, and the per-class scores are then averaged. Which average you pick changes the answer, so in scikit-learn the average parameter is a real decision:
macrotakes the plain mean of the per-class scores, so every class counts equally. A rare class that matters, such as a rare defect type, gets a voice. The cost is that a rare class with poor performance can drag the whole score down.weightedweights each class's score by how often that class appears in the true labels. Common classes dominate it.micropools the tp, fp and fn counts from all classes before dividing, so each sample-class pair counts once.Nonereturns one score per class with no averaging.
Here is how the choice plays out. Take a three-class model with per-class F1 of 0.95, 0.90 and 0.20, where the 0.20 class is 2% of rows. Its macro F1 is about 0.68. Its weighted F1 sits close to the two large classes. Report macro when the rare class is the reason the model exists, and weighted when overall throughput on common cases is what you are judged on. Databricks AutoML ranks classification runs by F1 unless you choose another primary_metric.
Checkpoint 4 of 10· Match them up
A multiclass defect classifier is scored with f1_score, and the result changes with the value passed to the average parameter. Match each setting to how it produces its result.
Tap a term, then the definition that fits it.
Macro gives every class equal weight, so a poorly served rare class pulls the score down. Weighted scales each class by its frequency, so common classes dominate. Micro pools the raw counts before dividing, and None skips averaging and returns the per-class scores.
“"macro" simply calculates the mean of the binary metrics, giving equal weight to each class.”Source: scikit-learn.org
4.ROC curve and ROC AUC: scoring the ranking, not one threshold
ROC AUC scores the model's scores rather than its labels, so no threshold has to be chosen. The ROC curve sweeps the threshold from high to low. At each step it plots the true positive rate (recall, tp / (tp + fn)) against the false positive rate (fp / (fp + tn), also called fall-out). ROC AUC is the area under that curve. A model that ranks every positive above every negative reaches 1.0, and a curve along the diagonal is no better than random. ROC AUC is higher-is-better.
A Databricks Unity Catalog example computes the curve with a Python UDTF over 14 scored rows, 6 positive and 8 negative. Each time the threshold passes a positive, the TPR goes up by 1/6. Each time it passes a negative, the FPR goes up by 1/8. The AUC for the whole curve is 0.786.
| threshold | true_positive_rate | false_positive_rate | auc |
|---|---|---|---|
| 0.82 | 0.5 | 0.0 | 0.786 |
| 0.78 | 0.5 | 0.125 | 0.786 |
| 0.52 | 0.833 | 0.375 | 0.786 |
| 0.31 | 1.0 | 0.625 | 0.786 |
| 0.03 | 1.0 | 1.0 | 0.786 |
Pass the probability of the positive class, not the predicted label. If you give roc_auc_score 0/1 predictions, the curve collapses to a single threshold. The Databricks getting-started notebook takes column 1 of predict_proba, then logs the result manually with mlflow.log_metric, because autologging does not record test AUC.
predicted_probs = model.predict_proba(X_test)
roc_auc = sklearn.metrics.roc_auc_score(y_test, predicted_probs[:,1])Checkpoint 5 of 10· Check yourself
A teammate scores a binary churn model with roc_auc_score(y_test, model.predict(X_test)) and wants a ROC AUC that reflects how well the model ranks customers. What should they change?
ROC AUC is computed from scores, not labels. Passing 0/1 predictions, at any threshold, collapses the curve to a single point. The positive-class probability column lets the curve sweep every threshold. Autologging does not record test AUC, which is why the Databricks notebook logs it manually.
“Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.”Source: scikit-learn.org
5.Precision-recall AUC when positives are very rare
When positives are very rare, the FPR denominator is huge, so a model can produce many false alarms and still keep a high ROC AUC. FPR divides fp by every negative. Precision divides fp only against the positives the model flagged. Take a screening set of 10,000 patients, 50 of them with the condition. Two models both find 40 of the 50 at their threshold. Model A raises 100 false alarms and model B raises 500.
| Model | tp | fp | Recall (TPR) | FPR = fp / 9,950 | Precision = tp / (tp + fp) |
|---|---|---|---|---|---|
| A | 40 | 100 | 0.8 | ≈0.010 | ≈0.29 |
| B | 40 | 500 | 0.8 | ≈0.050 | ≈0.074 |
Both FPRs sit close to zero, so the two ROC curves, and their AUCs, can look nearly the same. Precision drops from 0.29 to 0.074, which means that in model B only about 1 flagged patient in 14 actually has the condition. The precision-recall curve plots precision against recall across thresholds. Scikit-learn computes it with precision_recall_curve and summarises it with average_precision_score, and both take prediction scores, the same column you pass to roc_auc_score. Use PR-AUC, alongside or instead of ROC AUC, when the positive class is rare and false alarms carry a cost.
Two other ideas do not help here. Specificity, tn / (tn + fp), is just 1 − FPR, so it shares the same oversized denominator and never looks at how positives are handled. Recomputing ROC AUC on a different split changes the sample but still uses the metric that fails to separate the two models.
Checkpoint 6 of 10· Check yourself
Two screening models are scored on a set where only 0.5% of patients have the condition. Their ROC AUCs are 0.94 and 0.93, but clinicians suspect one raises far more false alarms. Which comparison would expose that difference?
Precision divides false positives by the rows the model flagged, not by the huge pool of negatives. So the precision-recall summary shows the false-alarm gap that FPR and ROC AUC hide, and like ROC AUC it takes prediction scores, not labels. Specificity is 1 − FPR, so it has the same oversized denominator, and a new split still uses the metric that failed to separate the models.
“average_precision_score(y_true, y_score, *) | Compute average precision (AP) from prediction scores.”Source: scikit-learn.org
Checkpoint 7 of 10· Exam question
A hospital builds a classifier to flag a rare condition that occurs in fewer than 1% of screened patients. Two candidate models have nearly identical ROC-AUC scores, but the clinical team wants a metric that will more clearly separate them given how rare the positive class is. Which evaluation should they add?
Correct answer: D — They should compute the precision-recall AUC, because with the positive class this rare, ROC-AUC can look similarly high despite very different false-positive counts, while precision and recall track that imbalance far more sharply.
- A. Overall accuracy is dominated by the huge majority of patients who do not have the condition, so a model that rarely flags anyone can still score very high accuracy while missing most true cases. It would not help distinguish two models on how well they handle the rare positive class.
- B. Specificity only describes correctness among patients without the condition and ignores how many true positive cases each model actually catches. Two models with very different sensitivity to the rare condition could still post similar specificity.
- C. Changing the train-test split addresses sampling variability rather than choosing a metric better suited to a rare positive class, and it does not change what ROC-AUC itself is measuring. The underlying insensitivity of ROC-AUC to extreme imbalance would remain.
- D. Because negatives vastly outnumber positives here, the false positive rate used by ROC-AUC can stay small even when the raw number of false positives is large relative to the few true positives, making ROC-AUC look deceptively similar across models. Precision and recall are computed directly against the small positive class, so the precision-recall curve reacts much more sharply to that difference.
6.Log loss: grading the probabilities themselves
Log loss scores the probability the model gave to the true class, so two models with identical accuracy can have very different log loss. It is also called logistic loss or cross-entropy loss. For each row it is the negative log-likelihood of the true label: −ln(p̂) when the row is positive and −ln(1 − p̂) when it is negative. These per-row values are averaged across the dataset. Log loss takes predict_proba output, not the thresholded labels, which is why the XGBoost tutorial passes y_pred_proba to log_loss and y_pred to accuracy_score.
The −ln curve is what makes log loss sensitive to calibration. A correct row at p̂ = 0.9 costs about 0.105. A wrong row at 0.4 costs 0.92, and the same wrong row at 0.01 costs 4.6. A model that is often wrong and overconfident collects large penalties that accuracy, precision and a confusion matrix never show, because all three see only which side of 0.5 a probability landed on. Cohen's kappa doesn't help either: it measures agreement between hard labels. Log loss is lower-is-better, with 0 for a model that gives the true class probability 1 on every row.
Checkpoint 8 of 10· Check yourself
Before picking one of two churn models, a reviewer finds that both give identical confusion matrices at a 0.5 threshold. On several customers who did churn, one model gave a churn probability of 0.01 and the other gave 0.4. Which metric would show a difference between the two models?
Log loss reads predict_proba output, so the 0.01 rows cost about 4.6 each against about 0.92 for the 0.4 rows. Kappa, balanced accuracy and F1 all read only the thresholded labels, and those labels are identical for the two models.
“mean | log loss | multi-class | predict_proba”Source: scikit-learn.org
Checkpoint 9 of 10· Exam question
Two churn-prediction models score identically on accuracy, but the team wants to pick the one whose predicted probabilities are best calibrated, penalizing predictions that are both wrong and overconfident. Which metric should the team compute on the held-out predictions to make this comparison?
Correct answer: B — Log loss should be computed, because it evaluates the predicted probability assigned to the true class and grows sharply for confident predictions that turn out to be wrong, not just for wrong predictions in general.
- A. A confusion matrix built from hard labels at one threshold throws away the underlying probability values entirely, so it cannot distinguish a confident wrong prediction from a barely-wrong one. That makes it the wrong tool for comparing probability calibration.
- B. Log loss takes the predicted probability assigned to the true class and applies a logarithmic penalty, so a confident but incorrect prediction is penalized far more heavily than a cautious, uncertain one. This directly rewards models whose probabilities are well calibrated rather than just their hard classifications.
- C. Precision at a fixed threshold still only looks at hard classification outcomes after the cutoff is applied, so it says nothing about how confident or overconfident the underlying probability estimates were. Two models can share identical precision while differing sharply in calibration.
- D. Cohen's kappa compares hard-label agreement against chance-level agreement, which is a useful check for classifier consistency but still collapses each prediction to a category before scoring. It does not capture how confidently the model made a given prediction.
7.Picking and optimising a classification metric on Databricks
A tuner or ranking needs to know which way each metric points. F1, precision, recall, accuracy, ROC AUC and PR-AUC are higher-is-better. Log loss is lower-is-better. The general rules, including scikit-learn's neg_ scorers, are in Metric direction: higher-is-better vs lower-is-better (neg_ scorers, minimized losses). On Databricks, the direction shows up in three places:
- AutoML's
classifyranks runs byprimary_metric, which acceptsf1(the default),log_loss,precision,accuracyorroc_auc. - An Optuna objective that returns AUC has to return
-roc_auc, because Optuna minimizes by default. A log-loss objective can be returned as it is. - Finding the best run with
mlflow.search_runssorts AUC withDESC. A log-loss column would be sorted ascending.
Checkpoint 10 of 10· Match them up
A team tunes and selects churn models on Databricks in four different ways. Match each setup to the handling that makes the best model come out on top.
Tap a term, then the definition that fits it.
Optuna minimizes by default, so a higher-is-better metric such as AUC has to be negated, while log loss is already lower-is-better and can be returned as it is. Sorting AUC with DESC puts the best run first, and AutoML ranks classification runs by F1 unless you set primary_metric to something else.
“# Negate the AUC because Optuna minimizes the objective by default return -roc_auc”Source: docs.databricks.com
Sources
(5)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/machine-learning/ai-runtime/examples/tutorials/sgc-xgboost-optuna-tuningOfficial docs
“y_pred = (y_pred_proba > 0.5).astype(int)”
↩︎ Confusion counts and why accuracy misleads on rare positives“conf_matrix = confusion_matrix(y_test, y_pred)”
↩︎ Confusion counts and why accuracy misleads on rare positives“test_logloss = log_loss(y_test, y_pred_proba)”
↩︎ Log loss: grading the probabilities themselves - 2.https://scikit-learn.org/stable/modules/model_evaluation.htmlSecondary source
“if the conventional accuracy is above chance only because the classifier takes advantage of an imbalanced test set”
↩︎ Confusion counts and why accuracy misleads on rare positives“Intuitively, precision is the ability of the classifier not to label as positive a sample that is negative”
↩︎ Precision, recall and F1 from the same counts“recall is the ability of the classifier to find all the positive samples.”
↩︎ Precision, recall and F1 from the same counts“F-measure is the weighted harmonic mean of precision and recall”
↩︎ Precision, recall and F1 from the same counts“Calculating recall (also called the true positive rate or the sensitivity) for each class:”
↩︎ Precision, recall and F1 from the same counts“by default only the positive label is evaluated, assuming by default that the positive class is labelled 1”
↩︎ Precision, recall and F1 from the same counts“"macro" simply calculates the mean of the binary metrics, giving equal weight to each class.”
↩︎ Averaging F1 across several classes“macro-averaging will over-emphasize the typically low performance on an infrequent class”
↩︎ Averaging F1 across several classes“each class’s score is weighted by its presence in the true data sample”
↩︎ Averaging F1 across several classes“this sums the dividends and divisors that make up the per-class metrics to calculate an overall quotient”
↩︎ Averaging F1 across several classes“Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.”
↩︎ ROC curve and ROC AUC: scoring the ranking, not one threshold“Calculating fall out (also called the false positive rate) for each class:”
↩︎ ROC curve and ROC AUC: scoring the ranking, not one threshold“Compute precision-recall pairs for different probability thresholds.”
↩︎ Precision-recall AUC when positives are very rare“Compute average precision (AP) from prediction scores.”
↩︎ Precision-recall AUC when positives are very rare“Calculating specificity (also called the true negative rate) for each class:”
↩︎ Precision-recall AUC when positives are very rare“Log loss, also called logistic regression loss or cross-entropy loss, is defined on probability estimates.”
↩︎ Log loss: grading the probabilities themselves“can be used to evaluate the probability outputs (predict_proba) of a classifier instead of its discrete predictions”
↩︎ Log loss: grading the probabilities themselves“Compute Cohen's kappa: a statistic that measures inter-annotator agreement.”
↩︎ Log loss: grading the probabilities themselves“Some metrics might require probability estimates of the positive class or non-thresholded decision values (as returned by decision_function on some classifiers).”
↩︎ Key concept“if the conventional accuracy is above chance only because the classifier takes advantage of an imbalanced test set”
↩︎ Exam trap 1“"macro" simply calculates the mean of the binary metrics, giving equal weight to each class.”
↩︎ Exam trap 2“Log loss, also called logistic regression loss or cross-entropy loss, is defined on probability estimates.”
↩︎ Exam trap 3“Compute precision-recall pairs for different probability thresholds.”
↩︎ Exam trap 4“The accuracy_score function computes the accuracy, either the fraction (default) or the count (normalize=False) of correct predictions.”
↩︎ Prediction“average_precision_score(y_true, y_score, *) | Compute average precision (AP) from prediction scores.”
↩︎ Checkpoint“the log loss per sample is the negative log-likelihood of the classifier given the true label”
↩︎ Prediction“mean | log loss | multi-class | predict_proba”
↩︎ Checkpoint - 3.
“This is useful for calculating metrics such as precision and recall.”
↩︎ Precision, recall and F1 from the same counts“Supported metrics for classification: “f1” (default), “log_loss”, “precision”, “accuracy”, “roc_auc””
↩︎ Averaging F1 across several classes“Metric used to evaluate and rank model performance.”
↩︎ Picking and optimising a classification metric on Databricks - 4.
“RETURNS TABLE (threshold DOUBLE, true_positive_rate DOUBLE, false_positive_rate DOUBLE, auc DOUBLE)”
↩︎ ROC curve and ROC AUC: scoring the ranking, not one threshold - 5.
“The AUC score on test data is not automatically logged, so log it manually”
↩︎ ROC curve and ROC AUC: scoring the ranking, not one threshold“roc_auc = sklearn.metrics.roc_auc_score(y_test, predicted_probs[:,1])”
↩︎ Precision-recall AUC when positives are very rare“Negate the AUC because Optuna minimizes the objective by default”
↩︎ Picking and optimising a classification metric on Databricks“order_by=['metrics.test_auc DESC', 'start_time DESC'],”
↩︎ Picking and optimising a classification metric on Databricks“# Negate the AUC because Optuna minimizes the objective by default return -roc_auc”
↩︎ Checkpoint