What you will be able to do
- Define a retraining policy as an alert whose condition reads a model monitor metric and whose action runs EXECUTE TASK on a retraining root task
- Gate a retraining graph so a new model version is promoted only after it passes evaluation
- Start retraining from newly arrived data with a stream-triggered root task instead of a polling schedule
- Explain why a newly created alert or task does nothing until it is resumed
- Troubleshoot a retraining policy that failed or went silent by reading alert history and auto-suspension state, and trace the signal to a warehouse capacity, container scalability or data dependency cause
Key concept
Retraining policy as an alert — A retraining policy is an alert that runs on a schedule. Its condition queries a model monitor metric against a threshold, and its action runs EXECUTE TASK to start the retraining graph whenever the condition returns rows.
1.A retraining policy is an alert whose action starts a task
A deployed model gets worse without anyone touching it. Input distributions drift, assumptions made at training time go stale, and upstream pipelines change what they deliver. A retraining policy turns that slow decay into a decision a machine makes: when a measured metric crosses a threshold, a new model is trained. Without a policy, someone has to notice a dashboard and kick off a run by hand. With one, the pipeline reacts within one evaluation interval.
The measurement comes from a model monitor. Model monitors (drift, stats and performance metrics) track one model version's drift, performance and volume from its stored inference data, and the monitor's metric functions are the only signal the policy reads. If that signal stops updating, the policy goes quiet without raising an error, which is why the monitor's health is part of troubleshooting retraining.
The policy itself is a Snowflake alert, which has two SQL parts. The condition is a SELECT, SHOW or CALL statement. For retraining, it selects from the monitor's drift or performance metric only where the value breaches your threshold, for example where a feature's drift is above the limit or where F1 has dropped below 0.80. If the statement returns rows, the alert runs its action. The action is any SQL statement. Snowflake alerts for model degradation and governance violations usually send a notification. A retraining policy instead runs EXECUTE TASK on the root task of the retraining graph, and that starts one run of the whole graph.
ALTER ALERT [ IF EXISTS ] <name> MODIFY CONDITION EXISTS (<condition>)
ALTER ALERT [ IF EXISTS ] <name> MODIFY ACTION <action>Other ways of starting the graph fail the policy because they react to something other than the metric. A cron schedule on the root task retrains every night, whether or not anything has drifted. A stream on the inference log fires every time new predictions land, which measures volume, not degradation. Only the alert ties the start of a run to the threshold. It does this from inside Snowflake, so no external service has to stay running to watch the metric.
| Design | Fires when | Fits a drift or degradation policy? |
|---|---|---|
| Alert: condition on monitor metric, action EXECUTE TASK | The condition query returns rows, meaning the threshold is breached | Yes |
| Root task with SCHEDULE (cron) | The clock reaches the scheduled time | No: it retrains whether or not drift occurred |
| Root task with WHEN SYSTEM$STREAM_HAS_DATA on the inference log | Any new rows are logged | No: it reacts to volume, not to the metric |
Checkpoint 1 of 8· Check yourself
A team wants its retraining task graph to run only when the model monitor shows F1 has dropped below 0.80, with no external service watching the metric. Which setup does this?
Only the alert ties the start of a run to the threshold: the condition returns rows only when F1 is below 0.80, and EXECUTE TASK on the root task starts one run of the whole graph. A cron schedule retrains whether or not anything has drifted, a stream trigger reacts to how much data arrives rather than to the metric, and a notification still leaves someone to start the run by hand.
“To run a single instance of a task graph, use EXECUTE TASK on the root task.”Source: docs.snowflake.com
Checkpoint 2 of 8· Exam question
A model monitor on a fraud-scoring model in Snowflake records drift metrics hourly. The team wants retraining to start automatically when the Jensen-Shannon drift of the txn_amount feature exceeds a threshold, with no always-on external service. Which design meets this requirement?
Correct answer: A — Create an alert whose condition queries the monitor's drift metric function for values above the threshold, and whose action runs EXECUTE TASK on the retraining root task.
- A. An alert evaluates its condition query on a schedule and runs a SQL action only when rows are returned, so checking the drift metric and executing the task gives a threshold-driven trigger entirely inside Snowflake.
- B. A stream-triggered task fires whenever new rows arrive, which is data arrival rather than measured drift, so it would retrain on every batch and ignore the threshold.
- C. A fixed nightly schedule retrains whether or not drift occurred and cannot respond to drift between runs, so it is a calendar policy rather than a drift-triggered one.
- D. Dynamic tables only materialize query results and cannot invoke stored procedures on refresh, so this cannot start training and it also ignores any drift threshold.
2.The retraining graph registers, evaluates, then promotes
An alert that starts retraining also has to protect production from a bad retrain. The retraining graph is a set of task graphs (DAGs) for ML pipelines steps: prepare training data, train, register the new model version, evaluate it, and finally move the production alias. The alias move goes last for a reason. As covered in Default versions, aliases (@prod), rollback and deprecation, inference that resolves @prod switches to whatever version the alias points at. A version that has been registered but not promoted is therefore harmless.
The gate comes from how task graphs handle failure. If a parent task does not run to completion, its child tasks are skipped. Make the alias-move task a child of the evaluation task, and make the evaluation task fail on purpose when the new version scores worse than the current one. A failed evaluation then means the alias is never moved, and @prod keeps serving the previous version. If a step has several parents, for example promotion waiting on both a metric check and a data-quality check, it waits until all of them have completed successfully.
Two details of how EXECUTE TASK behaves matter here. First, a run started this way executes only the child tasks that are resumed. Resume the whole graph with SYSTEM$TASK_DEPENDENTS_ENABLE on the root task, rather than resuming tasks one at a time and missing one. Second, a suspended child task does not block its successors. The graph treats a suspended child as if it had succeeded. If someone suspends the evaluation task "temporarily", the gate is gone: promotion runs straight after registration.
Checkpoint 3 of 8· Check yourself
During an incident, an engineer runs ALTER TASK … SUSPEND on the evaluation task of the retraining graph and leaves the other tasks resumed. A drift alert then runs EXECUTE TASK on the root task. What happens to the task that moves the @prod alias, which is the evaluation task's child?
The graph treats a suspended child task as if it had succeeded, so the tasks after it still run. Suspending the evaluation task turns the gate off, and the alias move goes ahead without any evaluation. Child tasks are skipped only when a parent fails to complete, not when a parent is suspended.
“When you suspend a child task, the task graph continues to run as though the child task had succeeded.”Source: docs.snowflake.com
3.Retraining when new labels arrive, without polling
Some policies react to fresh training data rather than to a metric. Suppose labeled outcomes land in labels_raw at irregular times. A root task scheduled every minute would mostly wake up to find nothing. A triggered task does better. Create a stream on labels_raw, then create the root task with a WHEN SYSTEM$STREAM_HAS_DATA condition on that stream and no SCHEDULE parameter. The task then runs when the stream has change records, and it uses no compute while waiting.
Two other approaches look similar but behave differently. An AFTER clause makes a task a child of another task in the same graph. It does not make a root task react to an outside loader, and the root task alone decides when a graph runs. ALTER TASK … RESUME changes a task's state but does not start a run. A task runs on its schedule, on its stream trigger, or when someone calls EXECUTE TASK. An alert whose action only resumes the root task has therefore enabled the graph without starting it.
Checkpoint 4 of 8· Match them up
Labeled outcomes land in labels_raw at irregular times. Match each mechanism to what it actually does to the retraining graph.
Tap a term, then the definition that fits it.
Only the root task decides when a graph runs: on a schedule, on a stream trigger, or when EXECUTE TASK is called. AFTER only orders child tasks inside the graph, and RESUME only enables a task. That is why a triggered root task fits new-data retraining, and why an alert's action has to be EXECUTE TASK, not RESUME.
“The root task defines when the task graph runs. Child tasks are executed in the order defined by the task graph.”Source: docs.snowflake.com
Checkpoint 5 of 8· Exam question
New labeled outcome rows are loaded into labels_raw by a pipeline at irregular times. A data scientist wants the retraining task graph to start only when new rows exist, and does not want to define a polling schedule on the root task. Which configuration is correct?
Correct answer: A — Create a stream on labels_raw, then create the root task with WHEN SYSTEM$STREAM_HAS_DATA on that stream and omit SCHEDULE so it runs as a triggered task.
- A. A triggered task has no SCHEDULE and uses a WHEN condition with SYSTEM$STREAM_HAS_DATA, so the graph starts only when the stream holds unconsumed changes.
- B. Defining a one-minute SCHEDULE makes the root task a polling scheduled task, which is exactly what the requirement rules out, and AFTER is only valid for child tasks rather than the root.
- C. There is no TRIGGER_ON_CHANGE task parameter; event-driven task execution is expressed through a stream condition in the WHEN clause.
- D. An alert is a scheduled poller, and resuming a task does not make it execute a run, so this keeps a polling schedule and does not start the graph on new data.
4.Turning the policy on: alerts and tasks start suspended
Alerts work the same way as tasks: a newly created alert is suspended and never evaluates its condition. A common failure looks like this. An engineer creates an alert that checks F1 every 15 minutes, the metric falls below the threshold, and days go by with no retraining. The alert has no recorded runs at all. Nothing is wrong with the threshold or the action. The alert was never resumed, so its condition was never checked. ALTER ALERT … RESUME makes a suspended alert active, and the retraining root task and its children need resuming too.
ALTER ALERT [ IF EXISTS ] <name> { RESUME | SUSPEND };
ALTER ALERT [ IF EXISTS ] <name> SET
[ WAREHOUSE = <string> ]
[ SCHEDULE = '{ <number> MINUTE | USING CRON <expr> <time_zone> }' ]The schedule syntax shows that a 15-minute interval is valid as '15 MINUTE', and cron is just the other option, so the schedule format is not why an alert stays silent. When a policy has never fired, check the alert's state first, then the root task's state, and only then the threshold logic.
Checkpoint 6 of 8· Check yourself
Before any labels arrive, an engineer creates the stream on labels_raw, a triggered root task (WHEN SYSTEM$STREAM_HAS_DATA, no SCHEDULE), and its train, register, evaluate and promote child tasks. What else must they do so that new labels run the whole retraining graph?
Every task is created suspended, so both the root task and its children must be resumed. SYSTEM$TASK_DEPENDENTS_ENABLE resumes them all in one call. If only the root task is resumed, the suspended children are treated as succeeded and do no work. A schedule is not needed, because the stream trigger is checked once the task is resumed.
“Snowflake creates all tasks in the SUSPENDED state. Verify the state of the task (or each task in a task graph) is RESUMED”Source: docs.snowflake.com
Checkpoint 7 of 8· Exam question
An engineer creates an alert that checks the model monitor's F1 metric every 15 minutes and should execute the retraining task when it falls below 0.80. Three days later retraining has never run, even though the metric has been below 0.80 for two days, and the alert has no entries in ALERT_HISTORY. What is the most likely cause?
Correct answer: C — The alert was created in its default suspended state and was never resumed, so it has not been evaluating; ALTER ALERT ... RESUME is needed to start it.
- A. A warehouse with a short auto-suspend resumes automatically when the alert needs it, so suspension between runs does not stop evaluations from happening.
- B. Alert schedules accept both interval and CRON forms, so the syntax choice is not what prevents any evaluations from being recorded.
- C. Alerts are created suspended and only evaluate after ALTER ALERT ... RESUME, which explains why ALERT_HISTORY is empty despite the metric being low for days.
- D. An alert action may run EXECUTE TASK with the alert owner's privileges, and granting the task to PUBLIC is not a requirement; this would also not explain the empty history.
5.Troubleshooting pipeline failures that surface at the retraining policy
A retraining policy can break in two places: the alert that decides to retrain, and the graph that alert starts. Troubleshooting pipeline failures begins by finding which one broke, because warehouse capacity, container scalability and data dependency problems each leave a different mark. ALERT_HISTORY returns one row per alert run, and its STATE column already sorts the cases. CONDITION_FALSE means the metric was within bounds and nothing should have happened. CONDITION_FAILED means the condition query itself errored. ACTION_FAILED means the threshold was breached but the EXECUTE TASK action did not succeed. In both failure states, SQL_ERROR_CODE and SQL_ERROR_MESSAGE give the reason.
CONDITION_FAILED usually points to a data dependency problem one layer above the graph. CREATE ALERT does not check identifiers or data types in the condition, so an alert that reads a renamed metric table is created without complaint and fails only when it runs. Test the condition query under the alert owner's role before you trust it. When the error traces back to a monitor or an upstream table, the stale-stream and suspended-monitor checks for data dependencies take over from there.
Silence is harder to read than an error. A policy that retrained twice last month and has done nothing for nine days may mean the metric recovered, or it may mean part of the chain suspended itself. Three objects in the chain can do that. An alert with SUSPEND_ALERT_AFTER_NUM_FAILURES set stops after that many consecutive failed or timed-out runs, and SHOW ALERTS then reports it as SUSPENDED with was_auto_suspended = TRUE. The retraining graph is suspended after 10 consecutive failures by default. A model monitor stops refreshing after five consecutive source-table failures, which freezes the metric the condition reads. Check all three before you conclude that drift has stopped. Resuming the alert also resets its failure counter to zero.
Warehouse capacity problems look different at the graph level than in a single step. Training, feature and evaluation tasks that run in parallel as children of one parent share a user-managed warehouse, so size it for all of them running at once, not for the heaviest single task. Serverless tasks stop scaling at the equivalent of an XXLARGE warehouse. A training step that needs more must be a user-managed task on a larger warehouse. Per-step timeouts and cancellations are diagnosed from task history, so this view only adds the concurrency and the size cap.
Container scalability failures leave no trace in the alert. When training runs on a compute pool, the alert shows TRIGGERED and the graph started, but the training job sits pending. That points to resource starvation on the pool's nodes, not to the policy. Dynamic tables that feed training follow the same rule from the other direction: a table reporting UPSTREAM_FAILED is never the root cause, because the root cause is always upstream.
Checkpoint 8 of 8· Check yourself
A drift alert started the retraining graph several times last month but has done nothing for nine days, even though the monitor dashboard still shows drift above the limit. SHOW ALERTS lists the alert as SUSPENDED with was_auto_suspended = TRUE. What has happened?
was_auto_suspended = TRUE means the alert hit its consecutive-failure limit. A manual suspend does not set that flag, and a condition that returns no rows is recorded as CONDITION_FALSE, not as a failure. Read ALERT_HISTORY for the failing runs' SQL_ERROR_MESSAGE, fix the cause, then run ALTER ALERT … RESUME.
“alerts are automatically suspended after the specified number of consecutive alert runs that either fail or time out”Source: docs.snowflake.com
Sources
(11)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.snowflake.com/en/developer-guide/snowflake-ml/model-registry/model-observabilityOfficial docs
“Model behavior can change over time due to input drift, stale training assumptions, and data pipeline issues”
↩︎ A retraining policy is an alert whose action starts a task“Model monitors automatically suspend refreshes when they encounter five consecutive refresh failures related to the source tables.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy - 2.
“If the statement returns one or more rows, the action for the alert is executed.”
↩︎ A retraining policy is an alert whose action starts a task“Specifies the SQL statement that should be executed if the condition returns one or more rows”
↩︎ A retraining policy is an alert whose action starts a task“RESUME makes a suspended alert active.”
↩︎ Turning the policy on: alerts and tasks start suspended“If the statement returns one or more rows, the action for the alert is executed.”
↩︎ Key concept - 3.
“To run a single instance of a task graph, use EXECUTE TASK on the root task.”
↩︎ A retraining policy is an alert whose action starts a task“When a task has multiple parents, the task waits for all preceding tasks to successfully complete before starting.”
↩︎ The retraining graph registers, evaluates, then promotes“When you run the root task, all resumed child tasks in the task graph are executed in the order defined by the task graph.”
↩︎ The retraining graph registers, evaluates, then promotes“When you suspend a child task, the task graph continues to run as though the child task had succeeded.”
↩︎ The retraining graph registers, evaluates, then promotes“The root task defines when the task graph runs.”
↩︎ Retraining when new labels arrive, without polling“By default, a task graph is suspended after 10 consecutive failures.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy“When tasks run in parallel on the same user-managed warehouse, the compute resources must be sized to handle the concurrent task runs.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy“When you suspend a child task, the task graph continues to run as though the child task had succeeded.”
↩︎ Exam trap 2“The root task defines when the task graph runs. Child tasks are executed in the order defined by the task graph.”
↩︎ Checkpoint - 4.https://docs.snowflake.com/en/user-guide/tasks-tsOfficial docs
“If a parent task failed to run to completion, any child tasks are skipped.”
↩︎ The retraining graph registers, evaluates, then promotes“Or verify that the task was manually executed using EXECUTE TASK.”
↩︎ Retraining when new labels arrive, without polling“Snowflake creates all tasks in the SUSPENDED state.”
↩︎ Turning the policy on: alerts and tasks start suspended“Or verify that the task was manually executed using EXECUTE TASK.”
↩︎ Exam trap 3“Snowflake creates all tasks in the SUSPENDED state. Verify the state of the task (or each task in a task graph) is RESUMED”
↩︎ Checkpoint - 5.
“Define the target stream using the WHEN clause. (Do not include the SCHEDULE parameter.)”
↩︎ Retraining when new labels arrive, without polling“This eliminates the need to poll a source frequently when the availability of new data is unpredictable.”
↩︎ Retraining when new labels arrive, without polling - 6.
“When you create an alert, the alert is suspended by default.”
↩︎ Turning the policy on: alerts and tasks start suspended“Instead, the failure occurs when the alert executes.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy“When you create an alert, the alert is suspended by default.”
↩︎ Exam trap 1 - 7.
“ACTION_FAILED: The condition was evaluated successfully, but the execution of the action failed.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy“CONDITION_FAILED: The evaluation of the condition failed.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy - 8.https://docs.snowflake.com/en/user-guide/alertsOfficial docs
“SHOW ALERTS reports the alert in the SUSPENDED state with was_auto_suspended = TRUE.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy“The counter resets to 0 when the alert run succeeds, or when the alert is manually suspended or resumed.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy“SHOW ALERTS reports the alert in the SUSPENDED state with was_auto_suspended = TRUE.”
↩︎ Exam trap 4“alerts are automatically suspended after the specified number of consecutive alert runs that either fail or time out”
↩︎ Checkpoint - 9.https://docs.snowflake.com/en/user-guide/tasks-introOfficial docs
“The maximum size for a serverless task run is equivalent to an XXLARGE warehouse.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy - 10.https://docs.snowflake.com/en/developer-guide/snowpark-container-services/troubleshootingOfficial docs
“This could be an indication of resource starvation on compute pool nodes.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy - 11.
“The root cause is always upstream.”
↩︎ Troubleshooting pipeline failures that surface at the retraining policy