AutoML Model and Feature Selection in Databricks

1.What AutoML automates, and what it hands back

Choosing a model normally means many manual decisions: which algorithm family to try, which hyperparameters to tune, how to handle missing values and how to compare the results fairly. Databricks AutoML automates that loop. You provide the dataset and say which kind of problem it is. AutoML then works through a fixed sequence:

1. It cleans and prepares your data. 2. It orchestrates distributed model training and hyperparameter tuning across multiple algorithms. 3. It finds the best model, using open source evaluation algorithms from scikit-learn, xgboost, LightGBM, Prophet and ARIMA. 4. It presents the results, together with generated source-code notebooks for each trial.

Step 2 is the model selection: AutoML doesn't commit to one algorithm, it compares several. Step 1 is where feature handling starts: AutoML decides how to treat each column. Step 4 makes both steps inspectable. The exam calls these generated notebooks "glassbox" notebooks. The documentation itself says the notebooks let you "review, reproduce, and modify the code."

You can start an experiment from a low-code UI (Experiments → the Classification, Regression or Forecasting card → Start training) or from the Python API (databricks.automl.classify, databricks.automl.regress). Both paths expose the same choices: dataset, target, evaluation metric and stopping conditions. AutoML relies on the libraries preinstalled in Databricks Runtime for Machine Learning. The docs warn that adding, removing, upgrading or downgrading those libraries causes run failures. Also, in Databricks Runtime 18.0 ML and above, AutoML is no longer included as a built-in library.

Sources1

2.Model selection: candidate algorithms, one ranking metric, a time budget

For classification and regression, AutoML draws its candidates from a fixed set of algorithm families. The decision tree, random forest, logistic regression and linear regression (stochastic gradient descent) models are based on scikit-learn. XGBoost and LightGBM come from their own libraries. Forecasting uses a different set (Prophet, Auto-ARIMA and, on serverless, DeepAR).

Every trial gets a score, and the score comes from a single primary metric. In the API this is primary_metric, described as the metric "used to evaluate and rank model performance". Classification defaults to f1, with log_loss, precision, accuracy and roc_auc as alternatives. Regression defaults to r2, with mae, rmse and mse as alternatives. This metric decides which model counts as "best". When a run completes, the top row of the runs table shows the best model based on the primary metric. Change the metric and you may get a different winner.

You can also narrow the field. exclude_frameworks (in the UI: excluding training frameworks under Advanced Configuration) removes whole families from the search. Its possible values are sklearn, lightgbm and xgboost, and the default is an empty list, meaning all frameworks are considered.

  exclude_cols: Optional[List[str]] = None,                         # :re[DBR] 10.3 ML and above
  exclude_frameworks: Optional[List[str]] = None,                   # :re[DBR] 10.3 ML and above
  feature_store_lookups: Optional[List[Dict]] = None,               # :re[DBR] 11.3 LTS ML and above
  imputers: Optional[Dict[str, Union[str, Dict[str, Any]]]] = None, # :re[DBR] 10.4 LTS ML and above

The last control is how long the search runs. The API parameter timeout_minutes defaults to 120 minutes, with a minimum of 5. The docs say "longer timeouts allow AutoML to run more trials and identify a model with better accuracy." The older max_trials parameter is not supported in Databricks Runtime 11.0 ML and above. On those runtimes the number of trials is not a stopping condition. For classification and regression, AutoML also stops early: it stops training and tuning when the validation metric is no longer improving.

Sources23

3.Feature selection: which columns AutoML uses and how it interprets them

AutoML's feature handling starts with the columns you let it see. In the UI, once you pick a table its schema appears, and you can choose which columns to train on. The API equivalent is exclude_cols, a "list of columns to ignore during AutoML calculations". Two columns are protected in the UI: you cannot remove the column selected as the prediction target or the time column used to split the data.

Next, AutoML decides how to read each remaining column. In Databricks Runtime 9.1 LTS ML and above, AutoML tries to detect whether a column has a semantic type different from the Spark or pandas data type in the schema, and treats it as the detected type. String or integer columns holding dates become timestamps. String columns holding numbers become numeric. From 10.1 ML, numeric columns that contain categorical IDs are treated as a categorical feature, and string columns containing English text are treated as a text feature. Detection is best effort.

Missing values are handled per column. Unless you say otherwise, AutoML picks an imputation method based on column type and content. You can override this with imputers (UI: the Impute with dropdown). The trade-off: if you specify a non-default imputation method, AutoML does not perform semantic type detection for that column.

AutoML also flags problem columns. With Databricks Runtime 10.1 ML and above, it shows warnings for potential dataset issues such as unsupported column types or high-cardinality columns, on the Warnings tab of the training page or experiment page. Use them to decide what to exclude on the next run. Databricks notes the warnings may not catch every issue.

Sources324

4.Adding features from feature tables

Feature selection can also mean adding columns, not only removing them. AutoML can augment the original input dataset with features from feature tables in Unity Catalog or in the legacy Workspace Feature Store. For classification and regression this requires Databricks Runtime 11.3 LTS ML and above. Forecasting requires 12.2 LTS ML and above.

In the UI, after configuring the experiment, click Join features (optional) and choose a feature table. For each primary key of the feature table, select the matching lookup key, which must be a column in your training dataset. For a time series feature table, you also select a timestamp lookup key, which must also be a column in the training dataset. Add more tables with Add another feature table.

In the API, the same join is a list of dictionaries passed to feature_store_lookups. Each dictionary needs table_name and lookup_key. The column names in lookup_key must be in the same order as the feature table's primary keys. timestamp_lookup_key is required only for time series feature tables, where it drives a point-in-time lookup.

 ?  = [
  {
     "table_name": "example.trip_pickup_features",
     "lookup_key": ["pickup_zip", "rounded_pickup_datetime"],
  },
  {
      "table_name": "example.trip_dropoff_features",
     "lookup_key": ["dropoff_zip", "rounded_dropoff_datetime"],
  }
]

Sources52

5.Reviewing the selection: generated notebooks and Shapley values

Because each trial's training code is generated, the model AutoML selects is never a black box. For classification and regression experiments, the data exploration notebook and the best trial's notebook are imported into your workspace automatically. Notebooks for the other trials are saved as MLflow artifacts on DBFS. For those trials, the notebook_path and notebook_url fields in the TrialInfo Python API are not set. You can import them through the AutoML experiment UI or the databricks.automl.import_notebook Python API. You can also open the MLflow run and find the notebook in the run page's Artifacts section, which you can download if your workspace admins allow artifact downloads. Forecasting is different: notebooks for all trials are imported.

When the experiment finishes, View notebook for best model opens the code behind the winner, and View data exploration notebook shows what AutoML saw in your data. Each run page also lists the trial's parameters, metrics and tags, plus its artifacts, including the model. The experiment's results, including the data exploration and training notebooks, are stored in a databricks_automl folder in the home folder of the user who ran it.

These notebooks also answer the feature question: which inputs actually drove the selected model? Regression and classification notebooks include code that computes Shapley values with the SHAP package. Shapley values come from game theory and estimate how much each feature contributes to a model's predictions. The calculation is memory-intensive, so it is off by default. To turn it on, go to the Feature importance section of a generated trial notebook, set shap_enabled = True, and re-run the notebook. On MLR 11.1 and below, SHAP plots are not generated if the dataset contains a datetime column.

Sources1

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.AutoML calculates SHAP feature importance automatically for every trial.Why is that wrong?

   The generated notebooks contain the SHAP code, but it doesn't run by default because it is memory-intensive. You set shap_enabled = True in the Feature importance section and re-run the notebook.

   Covered in Reviewing the selection: generated notebooks and Shapley values

2. 2.Every AutoML trial notebook for classification and regression is imported into the workspace.Why is that wrong?

   Only the data exploration notebook and the best trial's notebook are auto-imported. The others are MLflow artifacts that you import manually.

   Covered in Reviewing the selection: generated notebooks and Shapley values

3. 3.Setting a custom imputer for a column keeps AutoML's semantic type detection for that column.Why is that wrong?

   A non-default imputation method turns off semantic type detection for that column.

   Covered in Feature selection: which columns AutoML uses and how it interprets them

4. 4.On current runtimes you control how many models AutoML tries by setting max_trials.Why is that wrong?

   On Databricks Runtime 11.0 ML and above, max_trials is not supported and the trial count is not a stopping condition. Use timeout_minutes; early stopping also ends the search when the validation metric stops improving.

   Covered in Model selection: candidate algorithms, one ranking metric, a time budget

5. 5.lookup_key columns in feature_store_lookups can be listed in any order; AutoML matches them by name.Why is that wrong?

   The API reference requires the lookup_key column names to be in the same order as the feature table's primary keys.

   Covered in Adding features from feature tables