Cross-Validation vs Train-Validation Split: Tradeoffs

1.The single train-validation split and its hidden weakness

To tune hyperparameters or compare models, you need a score measured on data the model did not train on. The simplest way to get one is to carve the data into fixed pieces. You train on one piece, compare candidates on a validation piece, and keep a test piece for the final, untouched evaluation. Databricks AutoML does exactly this by default: when you don't specify a split strategy, it partitions the table into training, validation and test sets.

That split is cheap. Each candidate model is fit once and scored once. But it has two costs that are easy to miss. First, every row you reserve for validation is a row the model never learns from. With a 60/20/20 split, the model sees only 60% of your labelled data. Second, the validation score comes from one particular random set of rows. Reshuffle with a different seed and a different 20% lands in validation, so the score moves. On a large dataset that movement is small. On a small dataset it can be big enough to change which hyperparameter setting looks best.

Sources12

2.How k-fold cross-validation fixes it

Cross-validation replaces the fixed validation set with rotation. The training data is split into k folds. For each fold, a model is trained on the other k-1 folds and scored on that one. After k rounds, every training row has been used for validation exactly once and for fitting k-1 times. The reported score is the average of the k fold scores, and their spread (standard deviation) tells you how stable the estimate is. That fixes both weaknesses of the single split. No rows are permanently given up to validation, and the result no longer depends on one lucky or unlucky set of rows.

>>> from sklearn.model_selection import cross_val_score
>>> clf = svm.SVC(kernel='linear', C=1, random_state=42)
>>> scores = cross_val_score(clf, X, y, cv=5)
>>> scores
array([0.96, 1. , 0.96, 0.96, 1. ])

One thing cross-validation does not replace is the test set. The folds take over the validation role, but you still keep a test set that neither training nor tuning ever touches, so the final number reports generalization rather than how well you tuned. The same idea shows up in Databricks AutoML for forecasting. There the folds respect time order, which gives a robust evaluation across different segments of the series rather than a single cut.

Sources32

3.The price: k fits per candidate

Cross-validation's main downside is compute. A single split fits each candidate model once. k-fold cross-validation fits it k times, once per fold, so for the same number of candidates you pay roughly k times the training time. In a hyperparameter search that multiplier applies to every setting you try. A search that would fit 50 models with a single split fits 250 with 5-fold CV. For a small scikit-learn model that's trivial. For a large gradient-boosted ensemble or a model trained on a big table, it can be the difference between minutes and hours.

On Databricks, cross-validation usually runs inside the tuning loop. In the Hyperopt example below, the objective function scores each proposed value of C with cross_val_score and averages across folds. So every trial Hyperopt counts is really k model fits. The usual way to absorb the extra cost is parallelism. You distribute trials across Spark workers rather than drop cross-validation, but the total amount of training work still grows with k.

def objective(C):
    # Create a support vector classifier model
    clf = SVC(C=C)

    # Use the cross-validation accuracy to compare the models' performance
    accuracy = cross_val_score(clf, X, y).mean()

    # Hyperopt tries to minimize the objective function. A higher accuracy value means a better model, so you must return the negative accuracy.
    return {'loss': -accuracy, 'status': STATUS_OK}

    # Use the cross-validation accuracy to compare the models' performance
    accuracy =  ? (clf, X, y).mean()

Sources452

4.Choosing between them

Neither approach is always right. The choice depends on two questions: how scarce is the data, and how expensive is each fit? When data is small, cross-validation earns its cost. Holding out a fixed validation set would throw away a large share of a small dataset, and a single split's score would be noisy. When data is large and each fit is expensive, a single train-validation split is often good enough. A big validation set already gives a stable score, and multiplying training time by k buys little extra certainty. Spark MLlib offers both approaches as separate tuning classes, CrossValidator and TrainValidationSplit, so you choose based on that tradeoff.

Data shape matters too. Randomly shuffled folds assume that rows are interchangeable. With time-ordered data that's false: a random fold would train on the future and validate on the past. For forecasting, Databricks AutoML instead uses time-series cross-validation. The training window grows forward in time, and each round is validated on the period that comes next. You keep the variance-reducing benefit of multiple folds without leaking future information.

Sources632

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Cross-validation replaces the test set, so once you use k-fold CV you no longer need a held-out test set.Why is that wrong?

   Cross-validation replaces only the validation set. You still need a test set that tuning never touched for the final evaluation.

   Covered in How k-fold cross-validation fixes it

2. 2.Cross-validation gives a better estimate at roughly the same cost as a single split.Why is that wrong?

   Each candidate is trained once per fold, so compute grows about k-fold. The better estimate is paid for in training time.

   Covered in The price: k fits per candidate

3. 3.A single validation score is a stable measure of model quality, so a small gap between two settings means one is really better.Why is that wrong?

   A single split's score depends on which rows were randomly chosen for validation, so small gaps can flip when the split changes.

   Covered in The single train-validation split and its hidden weakness

4. 4.Shuffled k-fold cross-validation is always the safest choice, including for forecasting data.Why is that wrong?

   For time-ordered data, Databricks AutoML uses time-series cross-validation, which trains on earlier data and validates on later time points.

   Covered in Choosing between them