CertSafari
    Databricks Certified Machine Learning Associate· Lessons

    Domain 3 · Lesson 38/48

    Classification metrics: F1, ROC AUC, PR-AUC and log loss

    Use common classification metrics: F1, Log Loss, ROC/AUC, etc

    19 min read
    2.07% of exam
    5 sources
    Published 22 Oct 2026
    Docs as of 17 Oct 2026

    What you will be able to do

    • Compute precision, recall and F1 from confusion counts and explain why accuracy misleads on imbalanced labels
    • Choose between macro, weighted and micro averaging when F1 is extended to several classes
    • Read a ROC curve and ROC AUC from thresholded scores, and say when precision-recall AUC separates models that ROC AUC does not
    • Explain what log loss measures, why it punishes confident wrong predictions, and which direction is better for each metric

    Key concept

    What a classification metric consumes — Each classification metric reads one of three things: hard labels after a threshold (accuracy, precision, recall, F1), the ordering of scores across thresholds (ROC AUC, precision-recall AUC), or the probability given to the true class (log loss). What a metric reads decides what it can detect.

    1.Confusion counts and why accuracy misleads on rare positives

    A binary classifier turns a probability into a label with a threshold, and every metric in this section is computed from the four counts that come out. A row that is truly positive and predicted positive is a true positive (tp). A truly negative row predicted positive is a false positive (fp). A positive the model missed is a false negative (fn), and a correctly rejected negative is a true negative (tn). The confusion matrix just lays those four counts out in a grid.

    A Databricks tuning tutorial thresholds probabilities at 0.5, then computes accuracy from the labels and log loss from the probabilitiespython
        # Predict and evaluate on the held-out test set
        y_pred_proba = final_model.predict(dtest)
        y_pred = (y_pred_proba > 0.5).astype(int)
    
        accuracy = accuracy_score(y_test, y_pred)
        test_logloss = log_loss(y_test, y_pred_proba)

    Accuracy is (tp + tn) divided by every row, so the majority class dominates it. In the fraud set the 9,800 true negatives swamp the 200 rows that matter. A 98% score therefore tells the fraud team nothing until they see how many of the 200 frauds were caught. Scikit-learn's balanced accuracy exists to expose this: it falls back to chance level when plain accuracy beats chance only because the classifier takes advantage of an imbalanced test set. Rebalancing the training data is one fix, covered in Class imbalance: undersampling, oversampling/SMOTE, class weights. Whatever you do to the training data, report F1, precision and recall or PR-AUC on imbalanced labels, never accuracy alone.

    Checkpoint 1 of 10· Fill the gap

    This Databricks tutorial computes accuracy from labels thresholded at 0.5 and log loss on the same test set. Which argument completes the log_loss call?

        # Predict and evaluate on the held-out test set
        y_pred_proba = final_model.predict(dtest)
        y_pred = (y_pred_proba > 0.5).astype(int)
    
        accuracy = accuracy_score(y_test, y_pred)
        test_logloss = log_loss(y_test,  ? )

    Sources12

    2.Precision, recall and F1 from the same counts

    Precision and recall each ignore the true negatives, and that is why they work on rare positives. Precision is tp / (tp + fp): of the rows the model flagged, the fraction that really were positive. Recall is tp / (tp + fn): of the real positives, the fraction the model found. Recall also goes by true positive rate and sensitivity. Put intuitively, precision is the ability not to label a negative sample as positive, and recall is the ability to find all the positive samples.

    F1 is the harmonic mean of the two: F1 = 2 × precision × recall / (precision + recall). A harmonic mean is pulled towards the smaller value, so F1 rises only when both are good. Go back to the fraud set. The flag-nothing model has recall 0, which makes its F1 0 even though its accuracy is 98%. A model that flags 250 transactions, 150 of them real fraud, has precision 150/250 = 0.6 and recall 150/200 = 0.75. Its F1 is 2 × 0.6 × 0.75 / 1.35 ≈ 0.67. In scikit-learn's own small example, precision 1.0 and recall 0.5 give F1 ≈ 0.67, not the arithmetic mean of 0.75. F1 is the general F-measure with β = 1. Other values of β weight recall more or less heavily, and F1's best value is 1, its worst 0.

    The fraud set (200 positives out of 10,000) scored two ways
    ModeltpfpfnAccuracyPrecisionRecallF1
    Flags nothing002000.98undefined (scikit-learn returns 0 by default)0.00.0
    Flags 250, 150 correct150100500.9850.60.75≈0.67

    These metrics depend on which class you call positive. Scikit-learn's binary metrics score only the positive label, assumed to be 1 unless you set pos_label. Databricks AutoML's classify takes a pos_label argument for binary problems for the same reason, since it is needed to compute precision and recall. If the label is reversed, you are measuring how well the model finds legitimate transactions.

    Checkpoint 2 of 10· Check yourself

    In a payments table, fraud is stored as 0 and legitimate as 1. A data scientist calls f1_score(y_test, y_pred) with no other arguments and reports the 0.97 it returns as the model's fraud-detection F1. What does that 0.97 actually measure?

    Checkpoint 3 of 10· Exam question

    A data scientist at a payments company trains a classifier to flag fraudulent transactions in a dataset where only 2% of transactions are fraudulent. The model reports 98% accuracy, but the fraud team is skeptical that this reflects real detection ability. Which metric should the team report instead to better capture how well the model balances catching fraud against raising false alarms?

    Sources23

    3.Averaging F1 across several classes

    F1 is a binary metric. With several classes it is computed once per class, treating that class as positive, and the per-class scores are then averaged. Which average you pick changes the answer, so in scikit-learn the average parameter is a real decision:

    • macro takes the plain mean of the per-class scores, so every class counts equally. A rare class that matters, such as a rare defect type, gets a voice. The cost is that a rare class with poor performance can drag the whole score down.
    • weighted weights each class's score by how often that class appears in the true labels. Common classes dominate it.
    • micro pools the tp, fp and fn counts from all classes before dividing, so each sample-class pair counts once.
    • None returns one score per class with no averaging.

    Here is how the choice plays out. Take a three-class model with per-class F1 of 0.95, 0.90 and 0.20, where the 0.20 class is 2% of rows. Its macro F1 is about 0.68. Its weighted F1 sits close to the two large classes. Report macro when the rare class is the reason the model exists, and weighted when overall throughput on common cases is what you are judged on. Databricks AutoML ranks classification runs by F1 unless you choose another primary_metric.

    Checkpoint 4 of 10· Match them up

    A multiclass defect classifier is scored with f1_score, and the result changes with the value passed to the average parameter. Match each setting to how it produces its result.

    Tap a term, then the definition that fits it.

    Sources23

    4.ROC curve and ROC AUC: scoring the ranking, not one threshold

    ROC AUC scores the model's scores rather than its labels, so no threshold has to be chosen. The ROC curve sweeps the threshold from high to low. At each step it plots the true positive rate (recall, tp / (tp + fn)) against the false positive rate (fp / (fp + tn), also called fall-out). ROC AUC is the area under that curve. A model that ranks every positive above every negative reaches 1.0, and a curve along the diagonal is no better than random. ROC AUC is higher-is-better.

    A Databricks Unity Catalog example computes the curve with a Python UDTF over 14 scored rows, 6 positive and 8 negative. Each time the threshold passes a positive, the TPR goes up by 1/6. Each time it passes a negative, the FPR goes up by 1/8. The AUC for the whole curve is 0.786.

    Selected rows of the ROC curve from the UDTF example (6 positives, 8 negatives)
    thresholdtrue_positive_ratefalse_positive_rateauc
    0.820.50.00.786
    0.780.50.1250.786
    0.520.8330.3750.786
    0.311.00.6250.786
    0.031.01.00.786

    Pass the probability of the positive class, not the predicted label. If you give roc_auc_score 0/1 predictions, the curve collapses to a single threshold. The Databricks getting-started notebook takes column 1 of predict_proba, then logs the result manually with mlflow.log_metric, because autologging does not record test AUC.

    Test AUC from the positive-class probability columnpython
        predicted_probs = model.predict_proba(X_test)
        roc_auc = sklearn.metrics.roc_auc_score(y_test, predicted_probs[:,1])

    Checkpoint 5 of 10· Check yourself

    A teammate scores a binary churn model with roc_auc_score(y_test, model.predict(X_test)) and wants a ROC AUC that reflects how well the model ranks customers. What should they change?

    Sources245

    5.Precision-recall AUC when positives are very rare

    When positives are very rare, the FPR denominator is huge, so a model can produce many false alarms and still keep a high ROC AUC. FPR divides fp by every negative. Precision divides fp only against the positives the model flagged. Take a screening set of 10,000 patients, 50 of them with the condition. Two models both find 40 of the 50 at their threshold. Model A raises 100 false alarms and model B raises 500.

    Same recall, very different false alarms: FPR barely separates the models, precision does
    ModeltpfpRecall (TPR)FPR = fp / 9,950Precision = tp / (tp + fp)
    A401000.8≈0.010≈0.29
    B405000.8≈0.050≈0.074

    Both FPRs sit close to zero, so the two ROC curves, and their AUCs, can look nearly the same. Precision drops from 0.29 to 0.074, which means that in model B only about 1 flagged patient in 14 actually has the condition. The precision-recall curve plots precision against recall across thresholds. Scikit-learn computes it with precision_recall_curve and summarises it with average_precision_score, and both take prediction scores, the same column you pass to roc_auc_score. Use PR-AUC, alongside or instead of ROC AUC, when the positive class is rare and false alarms carry a cost.

    Two other ideas do not help here. Specificity, tn / (tn + fp), is just 1 − FPR, so it shares the same oversized denominator and never looks at how positives are handled. Recomputing ROC AUC on a different split changes the sample but still uses the metric that fails to separate the two models.

    Checkpoint 6 of 10· Check yourself

    Two screening models are scored on a set where only 0.5% of patients have the condition. Their ROC AUCs are 0.94 and 0.93, but clinicians suspect one raises far more false alarms. Which comparison would expose that difference?

    Checkpoint 7 of 10· Exam question

    A hospital builds a classifier to flag a rare condition that occurs in fewer than 1% of screened patients. Two candidate models have nearly identical ROC-AUC scores, but the clinical team wants a metric that will more clearly separate them given how rare the positive class is. Which evaluation should they add?

    Sources25

    6.Log loss: grading the probabilities themselves

    Log loss scores the probability the model gave to the true class, so two models with identical accuracy can have very different log loss. It is also called logistic loss or cross-entropy loss. For each row it is the negative log-likelihood of the true label: −ln(p̂) when the row is positive and −ln(1 − p̂) when it is negative. These per-row values are averaged across the dataset. Log loss takes predict_proba output, not the thresholded labels, which is why the XGBoost tutorial passes y_pred_proba to log_loss and y_pred to accuracy_score.

    The −ln curve is what makes log loss sensitive to calibration. A correct row at p̂ = 0.9 costs about 0.105. A wrong row at 0.4 costs 0.92, and the same wrong row at 0.01 costs 4.6. A model that is often wrong and overconfident collects large penalties that accuracy, precision and a confusion matrix never show, because all three see only which side of 0.5 a probability landed on. Cohen's kappa doesn't help either: it measures agreement between hard labels. Log loss is lower-is-better, with 0 for a model that gives the true class probability 1 on every row.

    Checkpoint 8 of 10· Check yourself

    Before picking one of two churn models, a reviewer finds that both give identical confusion matrices at a 0.5 threshold. On several customers who did churn, one model gave a churn probability of 0.01 and the other gave 0.4. Which metric would show a difference between the two models?

    Checkpoint 9 of 10· Exam question

    Two churn-prediction models score identically on accuracy, but the team wants to pick the one whose predicted probabilities are best calibrated, penalizing predictions that are both wrong and overconfident. Which metric should the team compute on the held-out predictions to make this comparison?

    Sources21

    7.Picking and optimising a classification metric on Databricks

    A tuner or ranking needs to know which way each metric points. F1, precision, recall, accuracy, ROC AUC and PR-AUC are higher-is-better. Log loss is lower-is-better. The general rules, including scikit-learn's neg_ scorers, are in Metric direction: higher-is-better vs lower-is-better (neg_ scorers, minimized losses). On Databricks, the direction shows up in three places:

    • AutoML's classify ranks runs by primary_metric, which accepts f1 (the default), log_loss, precision, accuracy or roc_auc.
    • An Optuna objective that returns AUC has to return -roc_auc, because Optuna minimizes by default. A log-loss objective can be returned as it is.
    • Finding the best run with mlflow.search_runs sorts AUC with DESC. A log-loss column would be sorted ascending.

    Checkpoint 10 of 10· Match them up

    A team tunes and selects churn models on Databricks in four different ways. Match each setup to the handling that makes the best model come out on top.

    Tap a term, then the definition that fits it.

    Sources35

    Sources

    (5)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 2.
      “if the conventional accuracy is above chance only because the classifier takes advantage of an imbalanced test set”
      ↩︎ Confusion counts and why accuracy misleads on rare positives
      “Intuitively, precision is the ability of the classifier not to label as positive a sample that is negative”
      ↩︎ Precision, recall and F1 from the same counts
      “recall is the ability of the classifier to find all the positive samples.”
      ↩︎ Precision, recall and F1 from the same counts
      “F-measure is the weighted harmonic mean of precision and recall”
      ↩︎ Precision, recall and F1 from the same counts
      “Calculating recall (also called the true positive rate or the sensitivity) for each class:”
      ↩︎ Precision, recall and F1 from the same counts
      “by default only the positive label is evaluated, assuming by default that the positive class is labelled 1”
      ↩︎ Precision, recall and F1 from the same counts
      “"macro" simply calculates the mean of the binary metrics, giving equal weight to each class.”
      ↩︎ Averaging F1 across several classes
      “macro-averaging will over-emphasize the typically low performance on an infrequent class”
      ↩︎ Averaging F1 across several classes
      “each class’s score is weighted by its presence in the true data sample”
      ↩︎ Averaging F1 across several classes
      “this sums the dividends and divisors that make up the per-class metrics to calculate an overall quotient”
      ↩︎ Averaging F1 across several classes
      “Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.”
      ↩︎ ROC curve and ROC AUC: scoring the ranking, not one threshold
      “Calculating fall out (also called the false positive rate) for each class:”
      ↩︎ ROC curve and ROC AUC: scoring the ranking, not one threshold
      “Compute precision-recall pairs for different probability thresholds.”
      ↩︎ Precision-recall AUC when positives are very rare
      “Compute average precision (AP) from prediction scores.”
      ↩︎ Precision-recall AUC when positives are very rare
      “Calculating specificity (also called the true negative rate) for each class:”
      ↩︎ Precision-recall AUC when positives are very rare
      “Log loss, also called logistic regression loss or cross-entropy loss, is defined on probability estimates.”
      ↩︎ Log loss: grading the probabilities themselves
      “can be used to evaluate the probability outputs (predict_proba) of a classifier instead of its discrete predictions”
      ↩︎ Log loss: grading the probabilities themselves
      “Compute Cohen's kappa: a statistic that measures inter-annotator agreement.”
      ↩︎ Log loss: grading the probabilities themselves
      “Some metrics might require probability estimates of the positive class or non-thresholded decision values (as returned by decision_function on some classifiers).”
      ↩︎ Key concept
      “if the conventional accuracy is above chance only because the classifier takes advantage of an imbalanced test set”
      ↩︎ Exam trap 1
      “"macro" simply calculates the mean of the binary metrics, giving equal weight to each class.”
      ↩︎ Exam trap 2
      “Log loss, also called logistic regression loss or cross-entropy loss, is defined on probability estimates.”
      ↩︎ Exam trap 3
      “Compute precision-recall pairs for different probability thresholds.”
      ↩︎ Exam trap 4
      “The accuracy_score function computes the accuracy, either the fraction (default) or the count (normalize=False) of correct predictions.”
      ↩︎ Prediction
      “average_precision_score(y_true, y_score, *) | Compute average precision (AP) from prediction scores.”
      ↩︎ Checkpoint
      “the log loss per sample is the negative log-likelihood of the classifier given the true label”
      ↩︎ Prediction
      “mean | log loss | multi-class | predict_proba”
      ↩︎ Checkpoint
    2. 3.
      “This is useful for calculating metrics such as precision and recall.”
      ↩︎ Precision, recall and F1 from the same counts
      “Supported metrics for classification: “f1” (default), “log_loss”, “precision”, “accuracy”, “roc_auc””
      ↩︎ Averaging F1 across several classes
      “Metric used to evaluate and rank model performance.”
      ↩︎ Picking and optimising a classification metric on Databricks
    3. 4.
      “RETURNS TABLE (threshold DOUBLE, true_positive_rate DOUBLE, false_positive_rate DOUBLE, auc DOUBLE)”
      ↩︎ ROC curve and ROC AUC: scoring the ranking, not one threshold
    4. 5.
      “The AUC score on test data is not automatically logged, so log it manually”
      ↩︎ ROC curve and ROC AUC: scoring the ranking, not one threshold
      “roc_auc = sklearn.metrics.roc_auc_score(y_test, predicted_probs[:,1])”
      ↩︎ Precision-recall AUC when positives are very rare
      “Negate the AUC because Optuna minimizes the objective by default”
      ↩︎ Picking and optimising a classification metric on Databricks
      “order_by=['metrics.test_auc DESC', 'start_time DESC'],”
      ↩︎ Picking and optimising a classification metric on Databricks
      “# Negate the AUC because Optuna minimizes the objective by default return -roc_auc”
      ↩︎ Checkpoint

    Ready to test yourself?

    Practise Databricks Certified Machine Learning Associate in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.