Counting Models in Grid Search with Cross-Validation

1.Cross-validation trains one new model per fold

To count the models in a tuning job, first be clear about what cross-validation does. In k-fold cross-validation, the training data is split into k parts called folds. One at a time, each fold is held out. A model is trained on the other k−1 folds and then scored on the held-out fold. The score reported at the end is the average of those k scores. So k-fold cross-validation does not train one model and score it k times. It trains k separate models on k different training sets, which is why the scikit-learn guide warns that the approach "can be computationally expensive."

The scikit-learn guide's own example shows this. It calls cross_val_score(clf, X, y, cv=5) on a linear SVM and describes the call as splitting the data, fitting a model and computing the score five times, with a different split each time. The result is an array of five scores, one for each fitted model. If you leave out cv, cross_val_score uses 5 folds by default in current scikit-learn versions.

The Databricks Hyperopt documentation uses the same unit. It says that in Hyperopt, "a trial generally corresponds to fitting one model on one setting of hyperparameters." Keep that unit in mind: one model is one hyperparameter setting fitted on one training set. A fold is a different training set, so it means another fit.

Sources12

2.A grid is a product, so the counts multiply

Grid search tries every combination of the values you list. If you list 3 values for one hyperparameter and 3 for another, the grid does not hold 3 + 3 = 6 candidates. It holds 3 × 3 = 9. Each extra hyperparameter multiplies the total again by the number of values listed for it.

The full formula is: models trained = (product of the number of values for each hyperparameter) × (number of folds). For the grid above, that is 3 × 3 × 5 = 45. In the Pipeline example the parameters belong to different steps (reduce_dim__ for PCA and clf__ for the classifier). That does not change the count, because every combination is still one candidate. The Databricks Hyperopt docs make the same equation between candidate settings and fits: max_evals is the "number of hyperparameter settings to try (the number of models to fit)". With cross-validation added, each of those settings costs k fits instead of one.

Sources13

3.Cross-validation inside a Hyperopt objective

The same multiplication applies when the search is not a grid. The Databricks example for parallelizing Hyperopt tunes the C parameter of an SVM. Each candidate's score comes from cross-validation, not from a single split. (Databricks notes that Hyperopt is no longer included in Databricks Runtime ML after 16.4 LTS ML. The counting logic is the same for any tuner.)

def objective(C):
    # Create a support vector classifier model
    clf = SVC(C=C)

    # Use the cross-validation accuracy to compare the models' performance
    accuracy = cross_val_score(clf, X, y).mean()

    # Hyperopt tries to minimize the objective function. A higher accuracy value means a better model, so you must return the negative accuracy.
    return {'loss': -accuracy, 'status': STATUS_OK}

The tuning loop sets how many candidates are tried, and the objective function sets how many models each candidate costs. The Databricks page tells you to "set max_evals to the maximum number of points in hyperparameter space to test". The example uses max_evals=16. Inside each trial, cross_val_score(clf, X, y) is called without cv, so it uses the default 5 folds. The run therefore tries 16 candidate values of C and trains up to 16 × 5 = 80 models. max_evals counts candidate settings, not individual fits.

argmin = fmin(
  fn=objective,
  space=search_space,
  algo=algo,
   ? =16)

Distributing the run does not change the count. When you pass SparkTrials to fmin(), its parallelism argument is the "number of models to fit and evaluate concurrently". That controls how many trials run at the same time, which changes how long the run takes. It does not change how many models are trained in total.

Sources4

4.Counting models on exam questions

Every counting question in this subdomain can be solved in three steps. First, count the candidate settings: multiply the number of values listed for each hyperparameter, or read max_evals for Hyperopt. Second, find k, the number of folds. Use the stated cv value or the tool's default: 5 for scikit-learn's GridSearchCV and cross_val_score, and 3 for Spark MLlib's CrossValidator (numFolds). Third, multiply the two. Add one only if the question asks about the final refit model. A TrainValidationSplit is the single-split case: it trains one model per setting instead of k.

Sources15

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The number of models trained equals the number of hyperparameter combinations in the grid.Why is that wrong?

   Each combination is cross-validated, so it is fitted once per fold. The total is combinations × folds, plus one refit in GridSearchCV's default setup.

   Covered in A grid is a product, so the counts multiply

2. 2.k-fold cross-validation trains one model and scores it on k different subsets.Why is that wrong?

   Each fold trains a new model on the other k−1 folds, and that model is scored only on its held-out fold. k folds means k separate fits.

   Covered in Cross-validation trains one new model per fold

3. 3.In Hyperopt, max_evals is always the total number of models trained.Why is that wrong?

   max_evals counts hyperparameter settings (trials). If the objective function runs cross-validation, each trial fits one model per fold, so the total is max_evals × folds.

   Covered in Cross-validation inside a Hyperopt objective

4. 4.Raising SparkTrials parallelism changes the number of models trained.Why is that wrong?

   parallelism only sets how many trials run at the same time. The total number of fits is still decided by the number of settings tried and the folds per setting.

   Covered in Cross-validation inside a Hyperopt objective