What you will be able to do
- Choose between scheduled and triggered retraining and justify the Databricks-recommended starting point
- Pick the right Lakeflow Jobs trigger type for a retraining workflow and tune table-update trigger timing
- Wire a data profiling drift or model-quality metric through a Databricks SQL alert and webhook to start the training workflow
Key concept
Retraining as a re-run of the production training pipeline — Automated retraining doesn't need a separate system. It re-runs the same production training pipeline, either on a schedule or when monitoring detects drift or degraded performance. The only design choice is what starts the run.
1.Scheduled first, triggered when needed
In the recommended Databricks MLOps architecture, the production training pipeline trains and tunes on production catalog tables, evaluates on held-out data, and registers a new model version in Unity Catalog. Something has to start that pipeline. The production-stage guidance says it can be started by code changes or by automated retraining jobs. Automated retraining comes in two forms: scheduled, where the job runs at a fixed interval on the newest data, and triggered, where a monitoring signal starts the run.
Scheduled retraining fits when new data arrives regularly. In that case you can create a scheduled job to run the model training code on the latest available data. Triggered retraining fits when the monitoring pipeline can spot problems. If the distribution of incoming data shifts significantly or model performance drops, automatic retraining and redeployment can restore performance with minimal human intervention. Automation has limits, though. If the retraining pipeline itself shows performance issues, the data scientist may need to go back to the development environment and experiment further. A new schedule won't fix a model whose features or algorithm no longer fit the problem.
Checkpoint 1 of 5· Check yourself
An automated retraining pipeline keeps producing models that perform poorly, whatever schedule or trigger starts it. What does the guidance say may be needed?
Changing what starts the run doesn't fix the pipeline itself. When the retraining pipeline has performance issues, the data scientist goes back to development to experiment further.
“the data scientist may need to return to the development environment for additional experimentation”Source: docs.databricks.com
Sources1
2.Lakeflow Jobs trigger types for retraining
Retraining runs as a Lakeflow Job, and the job's trigger decides when it starts. The recommended layout is a multitask workflow. The first task is the model training pipeline, followed by the model validation and model deployment tasks. The training task produces a model URI, and task values pass that URI to the validation task. Lakeflow Jobs offers several trigger types, and each one maps to a different retraining situation.
| Trigger type | Fires when | Retraining fit |
|---|---|---|
| Scheduled | A time-based schedule is reached | Periodic retraining on data that arrives regularly |
| Table update | Source tables are updated | Retrain when new training data lands, with no continuously running cluster |
| File arrival | New files arrive in a monitored Unity Catalog storage location | Retrain when raw files are dropped into a storage location |
| Model update | A Unity Catalog model is created, a model version is ready, or a model alias is set (Beta) | Start downstream validation or promotion once retraining registers a version |
| Continuous | A job run completes or fails, so another run starts | Keeps a job always running |
| None (manual) | Run now is clicked, or another orchestration tool starts the run | Includes runs started by an alert webhook or external orchestration tool |
A table update trigger is often the cleanest data-driven choice. It runs the job when new data is ready, without a continuously running cluster and without knowing which processes write to the table. It can watch several tables and fire when *any* or *all* of them change. Two advanced settings control timing. Minimum time between triggers waits a set interval after the previous run finishes. Wait after last change delays the run after an update, and each further update resets the timer. That suits data that arrives in batches. When the trigger fires, the dynamic value reference {{job.trigger.table_update.updated_tables}} hands the run a JSON list of tables updated since the last run.
Concurrency also matters. By default only one run of a job can be active at a time. Runs that exceed the configured maximum are skipped, not queued.
Checkpoint 2 of 5· Check yourself
Upstream ingestion writes to three source tables in a burst over several minutes. The retraining job should start once, after the whole batch has landed. Which table update trigger setting handles this?
Wait after last change delays the run after an update and resets the timer on each further update, so one run processes the whole batch. Any-table with no delay would fire on the first write.
“Additional table updates during this period reset the timer.”Source: docs.databricks.com
Checkpoint 3 of 5· Exam question
A partner system drops new labeled training batches into a Unity Catalog volume at unpredictable times throughout the day, sometimes hours apart and sometimes twice within a minute. The ML team wants the retraining job to start as soon as each batch lands, without polling on a fixed cadence. Which trigger should they configure on the retraining job?
Correct answer: A — A File arrival trigger pointed at the volume, so the job launches as soon as new objects appear in that storage location.
- A. A File arrival trigger watches a Unity Catalog storage location and launches a run when new objects show up there, which matches an unpredictable, event-driven batch drop with no need for polling.
- B. A five-minute schedule still polls on a fixed cadence, so it either wastes runs when nothing new has arrived or delays reaction to a batch that lands right after a poll, which does not match the stated requirement.
- C. A Continuous trigger chains a new run to the completion of the previous one, which produces a perpetual loop unrelated to whether new files have actually arrived in the volume.
- D. Raw files landing in a volume are not the same event as a Delta table version commit, so a Table update trigger would never fire unless something separately ingests those files into a monitored table.
3.From drift metric to retraining run
Triggered retraining needs a signal. Data profiling supplies it. When a profile runs on a table, such as an inference table holding model inputs and predictions, it creates or updates two metric tables: {output_schema}.{table_name}_profile_metrics and {output_schema}.{table_name}_drift_metrics. The profile metrics table holds per-column summary statistics. For InferenceLog analysis it also holds model accuracy metrics, but only if both label_col and prediction_col are provided. The drift metrics table tracks changes in distribution in two ways. Consecutive drift compares each window with the previous one. Baseline drift compares each window with a baseline table, and it is calculated only when you provide that table.
SELECT
window.start, column_name, count, num_nulls, distinct_count, frequent_items
FROM census_monitor_db.adult_census_profile_metrics
WHERE model_id = 1 -- Constrain to version 1
AND slice_key IS NULL -- look at aggregate metrics over the whole data
AND column_name = "income_predicted"
ORDER BY window.startA query like this is the starting point for a profile alert, which is an ordinary Databricks SQL alert. You write a SQL query on the profile metrics table or the drift metrics table, then create an alert on that query that is evaluated at a frequency you choose. The alert sends email by default. It can also post to a webhook, Slack, or PagerDuty. If the query uses parameters, the alert uses their default values, so check that those defaults express what you mean. The documentation names retraining directly: for InferenceLog analysis, a baseline-drift alert can indicate that the model should be retrained. The final link comes from the MLOps workflow. The SQL alert checks a drift or model-quality metric against a threshold, and its webhook destination triggers the training workflow.
Checkpoint 4 of 5· Put it in order
Put the steps for drift-triggered retraining in order.
- 1.Data profiling refreshes the profile and drift metrics tables for the inference table
- 2.The webhook triggers the model training workflow
- 3.Configure the alert to use a webhook destination
- 4.Create a Databricks SQL query on the drift metrics table
- 5.Create a Databricks SQL alert for that query, checking the metric against a threshold
The metric tables come first. The alert is built on a query over them, and its webhook destination is what starts the training job.
“The alert can be configured to use a webhook destination, which can subsequently trigger the training workflow.”Source: docs.databricks.com
Checkpoint 5 of 5· Check yourself
A team creates an InferenceLog profile without a baseline table, then builds an alert on baseline drift to trigger retraining. The alert never fires. Why?
With no baseline table there are no baseline-drift rows for the alert to evaluate. Only consecutive drift, window against previous window, is available.
“Baseline drift is only calculated if a baseline table is provided.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Best practice is to begin with drift-triggered retraining, because scheduled retraining wastes compute on models that haven't degraded.Why is that wrong?
Databricks recommends the opposite order. Start with scheduled, periodic retraining and move to triggered retraining only when needed.
Covered in Scheduled first, triggered when needed
2.A parameterized profile alert query is evaluated with whatever parameter values were last used in the SQL editor.Why is that wrong?
The alert always uses the query's default parameter values, so those defaults have to match the condition you want to alert on.
Covered in From drift metric to retraining run
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“This pipeline can be triggered by code changes or by automated retraining jobs.”
↩︎ Scheduled first, triggered when needed“you can create a scheduled job to run the model training code on the latest available data”
↩︎ Scheduled first, triggered when needed“automatic retraining and redeployment can boost model performance with minimal human intervention”
↩︎ Scheduled first, triggered when needed“the first task is the model training pipeline, followed by model validation and model deployment tasks”
↩︎ Lakeflow Jobs trigger types for retraining“This can be achieved through a SQL alert to check whether a metric is anomalous”
↩︎ From drift metric to retraining run“This architecture supports automatic retraining using the same model training pipeline above.”
↩︎ Key concept“Databricks recommends beginning with scheduled, periodic retraining and moving to triggered retraining when needed.”
↩︎ Exam trap 1“Databricks recommends beginning with scheduled, periodic retraining and moving to triggered retraining when needed.”
↩︎ Prediction“the data scientist may need to return to the development environment for additional experimentation”
↩︎ Checkpoint“The alert can be configured to use a webhook destination, which can subsequently trigger the training workflow.”
↩︎ Checkpoint - 2.
“When selecting multiple tables, you can specify if a job is triggered when any or all of the tables are updated.”
↩︎ Lakeflow Jobs trigger types for retraining“Additional table updates during this period reset the timer.”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/jobs/triggersOfficial docs
“By default, only a single run of a job can be active at a time.”
↩︎ Lakeflow Jobs trigger types for retraining“Runs are skipped when they exceed the configured max concurrency for a job.”
↩︎ Lakeflow Jobs trigger types for retraining - 4.
“Model quality is calculated if both label_col and prediction_col are provided.”
↩︎ From drift metric to retraining run“Baseline drift is only calculated if a baseline table is provided.”
↩︎ Checkpoint - 5.
“for InferenceLog analysis, to indicate that the model should be retrained.”
↩︎ From drift metric to retraining run“If the query uses parameters, the alert is based on the default values for those parameters.”
↩︎ Exam trap 2