Grid, random and Bayesian hyperparameter search compared

1.What a hyperparameter search actually does

A hyperparameter search runs the same loop again and again. It picks a hyperparameter setting, trains a model with it, scores that model, and keeps the best one. Hyperopt calls each pass of the loop a trial, and a trial generally means fitting one model on one setting of hyperparameters. That makes the budget easy to count: if you ask for 16 evaluations, you pay for 16 model fits. In Hyperopt that budget is the max_evals argument, described as the number of hyperparameter settings to try, which is also the number of models to fit.

The three methods differ in one place only: how the next setting is chosen. Grid search walks through a fixed list of combinations. Random search samples from the space. Bayesian search looks at the results so far before choosing. Training, scoring and the cost of each trial work the same way in all three. The rest of this page goes through the choice rule of each one.

Sources1

2.Grid search: every combination of a fixed list

In grid search you list the candidate values for each hyperparameter, and the search evaluates the combinations of those lists. The scikit-learn cross-validation guide notes that the best parameters can be determined by grid search techniques. Its pipeline documentation shows GridSearchCV tuning a whole pipeline. In that example the grid covers which dimensionality-reduction step to use, which classifier to use, and the classifier's C value.

>>> param_grid = dict(reduce_dim=['passthrough', PCA(5), PCA(10)],
...                   clf=[SVC(), LogisticRegression()],
...                   clf__C=[0.1, 10, 100])
>>> grid_search = GridSearchCV(pipe, param_grid=param_grid)

The Spark MLlib equivalents are CrossValidator and TrainValidationSplit. Older Databricks runtimes logged their hyperparameters and metrics to MLflow automatically, but that MLlib automated tracking is deprecated and is disabled by default from Databricks Runtime 10.4 LTS ML. A grid's main weakness is that it only tests the values you wrote down. It cannot try anything between C=10 and C=100.

Sources23

3.Random search: sampling from distributions

Random search does not use a list of values. You describe each hyperparameter as a distribution or a categorical choice, and the search draws settings from it. In Hyperopt, hyperopt.rand.suggest selects random search, which Databricks describes as a non-adaptive approach that samples over the search space. The space can mix categorical options, such as which algorithm to use, with numeric distributions such as uniform and log.

    param_space={
        "lr": tune.loguniform(1e-5, 1e-3),
        "lora_r": tune.choice([8, 16, 32]),
        "lora_alpha_ratio": tune.choice([1, 2]),
        "lora_dropout": tune.uniform(0.0, 0.1),
        "weight_decay": tune.choice([0.0, 0.01]),
        "batch_size": tune.choice([4, 8]),
    },

Because lr comes from a log-uniform range, two trials almost never share the same learning rate, so the search can land on values a grid would never list. Random sampling is also the default behaviour in Ray Tune. The Databricks example notes that if you want configurations to be *chosen* rather than sampled randomly, you pass TuneConfig a search_alg such as Optuna.

Sources14

4.Bayesian search: letting past trials pick the next one

Bayesian search keeps a record of which settings produced which losses and uses it to choose the next trial. In Hyperopt this is hyperopt.tpe.suggest, the Tree of Parzen Estimators. Databricks' best-practices page says Bayesian approaches can be much more efficient than grid search and random search, so with TPE you can explore more hyperparameters and wider ranges. The same page adds that using domain knowledge to narrow the search domain still improves the results. TPE is not limited to Hyperopt either: Optuna's MlflowSparkStudy uses optuna.samplers.TPESampler as its default sampler.

This is the trade-off between parallelism and adaptivity. Running more trials at once finishes a fixed budget sooner. Running fewer at once gives each new proposal more evidence to work from. Random search has nothing to learn from earlier trials, so this trade-off does not apply to it. Also, do not expect the loss to fall with every trial. Hyperopt's search is stochastic, so the loss usually does not decrease monotonically, although these methods often find the best hyperparameters faster than other methods.

Sources56

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.With TPE, running as many trials in parallel as possible always gives the best result.Why is that wrong?

   TPE adapts using past results. For a fixed max_evals, more parallelism is faster, but lower parallelism may give better results because each proposal has more completed trials to learn from.

   Covered in Bayesian search: letting past trials pick the next one

2. 2.If the loss goes up between consecutive Hyperopt trials, the search is broken.Why is that wrong?

   Hyperopt's algorithms are stochastic, so the loss is not expected to fall with every run.

   Covered in Bayesian search: letting past trials pick the next one