CertSafari
    Snowflake SnowPro Advanced: MLOps Engineer (MLA-B01)· Lessons

    Domain 4 · Lesson 14/17

    Automated retraining policies with alerts and triggered tasks

    Implement retraining and troubleshooting.

    16 min read
    7.34% of exam
    11 sources
    Published 21 Oct 2026
    Docs as of 18 Oct 2026

    What you will be able to do

    • Define a retraining policy as an alert whose condition reads a model monitor metric and whose action runs EXECUTE TASK on a retraining root task
    • Gate a retraining graph so a new model version is promoted only after it passes evaluation
    • Start retraining from newly arrived data with a stream-triggered root task instead of a polling schedule
    • Explain why a newly created alert or task does nothing until it is resumed
    • Troubleshoot a retraining policy that failed or went silent by reading alert history and auto-suspension state, and trace the signal to a warehouse capacity, container scalability or data dependency cause

    Key concept

    Retraining policy as an alert — A retraining policy is an alert that runs on a schedule. Its condition queries a model monitor metric against a threshold, and its action runs EXECUTE TASK to start the retraining graph whenever the condition returns rows.

    1.A retraining policy is an alert whose action starts a task

    A deployed model gets worse without anyone touching it. Input distributions drift, assumptions made at training time go stale, and upstream pipelines change what they deliver. A retraining policy turns that slow decay into a decision a machine makes: when a measured metric crosses a threshold, a new model is trained. Without a policy, someone has to notice a dashboard and kick off a run by hand. With one, the pipeline reacts within one evaluation interval.

    The measurement comes from a model monitor. Model monitors (drift, stats and performance metrics) track one model version's drift, performance and volume from its stored inference data, and the monitor's metric functions are the only signal the policy reads. If that signal stops updating, the policy goes quiet without raising an error, which is why the monitor's health is part of troubleshooting retraining.

    The policy itself is a Snowflake alert, which has two SQL parts. The condition is a SELECT, SHOW or CALL statement. For retraining, it selects from the monitor's drift or performance metric only where the value breaches your threshold, for example where a feature's drift is above the limit or where F1 has dropped below 0.80. If the statement returns rows, the alert runs its action. The action is any SQL statement. Snowflake alerts for model degradation and governance violations usually send a notification. A retraining policy instead runs EXECUTE TASK on the root task of the retraining graph, and that starts one run of the whole graph.

    An alert is a condition plus an action, and each can be changed separatelysql
    ALTER ALERT [ IF EXISTS ] <name> MODIFY CONDITION EXISTS (<condition>)
    
    ALTER ALERT [ IF EXISTS ] <name> MODIFY ACTION <action>

    Other ways of starting the graph fail the policy because they react to something other than the metric. A cron schedule on the root task retrains every night, whether or not anything has drifted. A stream on the inference log fires every time new predictions land, which measures volume, not degradation. Only the alert ties the start of a run to the threshold. It does this from inside Snowflake, so no external service has to stay running to watch the metric.

    What actually starts retraining under each design
    DesignFires whenFits a drift or degradation policy?
    Alert: condition on monitor metric, action EXECUTE TASKThe condition query returns rows, meaning the threshold is breachedYes
    Root task with SCHEDULE (cron)The clock reaches the scheduled timeNo: it retrains whether or not drift occurred
    Root task with WHEN SYSTEM$STREAM_HAS_DATA on the inference logAny new rows are loggedNo: it reacts to volume, not to the metric

    Checkpoint 1 of 8· Check yourself

    A team wants its retraining task graph to run only when the model monitor shows F1 has dropped below 0.80, with no external service watching the metric. Which setup does this?

    Checkpoint 2 of 8· Exam question

    A model monitor on a fraud-scoring model in Snowflake records drift metrics hourly. The team wants retraining to start automatically when the Jensen-Shannon drift of the txn_amount feature exceeds a threshold, with no always-on external service. Which design meets this requirement?

    Sources123

    2.The retraining graph registers, evaluates, then promotes

    An alert that starts retraining also has to protect production from a bad retrain. The retraining graph is a set of task graphs (DAGs) for ML pipelines steps: prepare training data, train, register the new model version, evaluate it, and finally move the production alias. The alias move goes last for a reason. As covered in Default versions, aliases (@prod), rollback and deprecation, inference that resolves @prod switches to whatever version the alias points at. A version that has been registered but not promoted is therefore harmless.

    The gate comes from how task graphs handle failure. If a parent task does not run to completion, its child tasks are skipped. Make the alias-move task a child of the evaluation task, and make the evaluation task fail on purpose when the new version scores worse than the current one. A failed evaluation then means the alias is never moved, and @prod keeps serving the previous version. If a step has several parents, for example promotion waiting on both a metric check and a data-quality check, it waits until all of them have completed successfully.

    Two details of how EXECUTE TASK behaves matter here. First, a run started this way executes only the child tasks that are resumed. Resume the whole graph with SYSTEM$TASK_DEPENDENTS_ENABLE on the root task, rather than resuming tasks one at a time and missing one. Second, a suspended child task does not block its successors. The graph treats a suspended child as if it had succeeded. If someone suspends the evaluation task "temporarily", the gate is gone: promotion runs straight after registration.

    Checkpoint 3 of 8· Check yourself

    During an incident, an engineer runs ALTER TASK … SUSPEND on the evaluation task of the retraining graph and leaves the other tasks resumed. A drift alert then runs EXECUTE TASK on the root task. What happens to the task that moves the @prod alias, which is the evaluation task's child?

    Sources43

    3.Retraining when new labels arrive, without polling

    Some policies react to fresh training data rather than to a metric. Suppose labeled outcomes land in labels_raw at irregular times. A root task scheduled every minute would mostly wake up to find nothing. A triggered task does better. Create a stream on labels_raw, then create the root task with a WHEN SYSTEM$STREAM_HAS_DATA condition on that stream and no SCHEDULE parameter. The task then runs when the stream has change records, and it uses no compute while waiting.

    Two other approaches look similar but behave differently. An AFTER clause makes a task a child of another task in the same graph. It does not make a root task react to an outside loader, and the root task alone decides when a graph runs. ALTER TASK … RESUME changes a task's state but does not start a run. A task runs on its schedule, on its stream trigger, or when someone calls EXECUTE TASK. An alert whose action only resumes the root task has therefore enabled the graph without starting it.

    Checkpoint 4 of 8· Match them up

    Labeled outcomes land in labels_raw at irregular times. Match each mechanism to what it actually does to the retraining graph.

    Tap a term, then the definition that fits it.

    Checkpoint 5 of 8· Exam question

    New labeled outcome rows are loaded into labels_raw by a pipeline at irregular times. A data scientist wants the retraining task graph to start only when new rows exist, and does not want to define a polling schedule on the root task. Which configuration is correct?

    Sources534

    4.Turning the policy on: alerts and tasks start suspended

    Alerts work the same way as tasks: a newly created alert is suspended and never evaluates its condition. A common failure looks like this. An engineer creates an alert that checks F1 every 15 minutes, the metric falls below the threshold, and days go by with no retraining. The alert has no recorded runs at all. Nothing is wrong with the threshold or the action. The alert was never resumed, so its condition was never checked. ALTER ALERT … RESUME makes a suspended alert active, and the retraining root task and its children need resuming too.

    Resuming an alert, and the schedule forms it accepts: an interval in minutes or a cron expressionsql
    ALTER ALERT [ IF EXISTS ] <name> { RESUME | SUSPEND };
    
    ALTER ALERT [ IF EXISTS ] <name> SET
      [ WAREHOUSE = <string> ]
      [ SCHEDULE = '{ <number> MINUTE | USING CRON <expr> <time_zone> }' ]

    The schedule syntax shows that a 15-minute interval is valid as '15 MINUTE', and cron is just the other option, so the schedule format is not why an alert stays silent. When a policy has never fired, check the alert's state first, then the root task's state, and only then the threshold logic.

    Checkpoint 6 of 8· Check yourself

    Before any labels arrive, an engineer creates the stream on labels_raw, a triggered root task (WHEN SYSTEM$STREAM_HAS_DATA, no SCHEDULE), and its train, register, evaluate and promote child tasks. What else must they do so that new labels run the whole retraining graph?

    Checkpoint 7 of 8· Exam question

    An engineer creates an alert that checks the model monitor's F1 metric every 15 minutes and should execute the retraining task when it falls below 0.80. Three days later retraining has never run, even though the metric has been below 0.80 for two days, and the alert has no entries in ALERT_HISTORY. What is the most likely cause?

    Sources624

    5.Troubleshooting pipeline failures that surface at the retraining policy

    A retraining policy can break in two places: the alert that decides to retrain, and the graph that alert starts. Troubleshooting pipeline failures begins by finding which one broke, because warehouse capacity, container scalability and data dependency problems each leave a different mark. ALERT_HISTORY returns one row per alert run, and its STATE column already sorts the cases. CONDITION_FALSE means the metric was within bounds and nothing should have happened. CONDITION_FAILED means the condition query itself errored. ACTION_FAILED means the threshold was breached but the EXECUTE TASK action did not succeed. In both failure states, SQL_ERROR_CODE and SQL_ERROR_MESSAGE give the reason.

    CONDITION_FAILED usually points to a data dependency problem one layer above the graph. CREATE ALERT does not check identifiers or data types in the condition, so an alert that reads a renamed metric table is created without complaint and fails only when it runs. Test the condition query under the alert owner's role before you trust it. When the error traces back to a monitor or an upstream table, the stale-stream and suspended-monitor checks for data dependencies take over from there.

    Silence is harder to read than an error. A policy that retrained twice last month and has done nothing for nine days may mean the metric recovered, or it may mean part of the chain suspended itself. Three objects in the chain can do that. An alert with SUSPEND_ALERT_AFTER_NUM_FAILURES set stops after that many consecutive failed or timed-out runs, and SHOW ALERTS then reports it as SUSPENDED with was_auto_suspended = TRUE. The retraining graph is suspended after 10 consecutive failures by default. A model monitor stops refreshing after five consecutive source-table failures, which freezes the metric the condition reads. Check all three before you conclude that drift has stopped. Resuming the alert also resets its failure counter to zero.

    Warehouse capacity problems look different at the graph level than in a single step. Training, feature and evaluation tasks that run in parallel as children of one parent share a user-managed warehouse, so size it for all of them running at once, not for the heaviest single task. Serverless tasks stop scaling at the equivalent of an XXLARGE warehouse. A training step that needs more must be a user-managed task on a larger warehouse. Per-step timeouts and cancellations are diagnosed from task history, so this view only adds the concurrency and the size cap.

    Container scalability failures leave no trace in the alert. When training runs on a compute pool, the alert shows TRIGGERED and the graph started, but the training job sits pending. That points to resource starvation on the pool's nodes, not to the policy. Dynamic tables that feed training follow the same rule from the other direction: a table reporting UPSTREAM_FAILED is never the root cause, because the root cause is always upstream.

    Checkpoint 8 of 8· Check yourself

    A drift alert started the retraining graph several times last month but has done nothing for nine days, even though the monitor dashboard still shows drift above the limit. SHOW ALERTS lists the alert as SUSPENDED with was_auto_suspended = TRUE. What has happened?

    Sources7683191011

    Sources

    (11)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Model behavior can change over time due to input drift, stale training assumptions, and data pipeline issues”
      ↩︎ A retraining policy is an alert whose action starts a task
      “Model monitors automatically suspend refreshes when they encounter five consecutive refresh failures related to the source tables.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy
    2. 2.
      “If the statement returns one or more rows, the action for the alert is executed.”
      ↩︎ A retraining policy is an alert whose action starts a task
      “Specifies the SQL statement that should be executed if the condition returns one or more rows”
      ↩︎ A retraining policy is an alert whose action starts a task
      “RESUME makes a suspended alert active.”
      ↩︎ Turning the policy on: alerts and tasks start suspended
      “If the statement returns one or more rows, the action for the alert is executed.”
      ↩︎ Key concept
    3. 3.
      “To run a single instance of a task graph, use EXECUTE TASK on the root task.”
      ↩︎ A retraining policy is an alert whose action starts a task
      “When a task has multiple parents, the task waits for all preceding tasks to successfully complete before starting.”
      ↩︎ The retraining graph registers, evaluates, then promotes
      “When you run the root task, all resumed child tasks in the task graph are executed in the order defined by the task graph.”
      ↩︎ The retraining graph registers, evaluates, then promotes
      “When you suspend a child task, the task graph continues to run as though the child task had succeeded.”
      ↩︎ The retraining graph registers, evaluates, then promotes
      “The root task defines when the task graph runs.”
      ↩︎ Retraining when new labels arrive, without polling
      “By default, a task graph is suspended after 10 consecutive failures.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy
      “When tasks run in parallel on the same user-managed warehouse, the compute resources must be sized to handle the concurrent task runs.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy
      “When you suspend a child task, the task graph continues to run as though the child task had succeeded.”
      ↩︎ Exam trap 2
      “The root task defines when the task graph runs. Child tasks are executed in the order defined by the task graph.”
      ↩︎ Checkpoint
    4. 4.
      “If a parent task failed to run to completion, any child tasks are skipped.”
      ↩︎ The retraining graph registers, evaluates, then promotes
      “Or verify that the task was manually executed using EXECUTE TASK.”
      ↩︎ Retraining when new labels arrive, without polling
      “Snowflake creates all tasks in the SUSPENDED state.”
      ↩︎ Turning the policy on: alerts and tasks start suspended
      “Or verify that the task was manually executed using EXECUTE TASK.”
      ↩︎ Exam trap 3
      “Snowflake creates all tasks in the SUSPENDED state. Verify the state of the task (or each task in a task graph) is RESUMED”
      ↩︎ Checkpoint
    5. 5.
      “Define the target stream using the WHEN clause. (Do not include the SCHEDULE parameter.)”
      ↩︎ Retraining when new labels arrive, without polling
      “This eliminates the need to poll a source frequently when the availability of new data is unpredictable.”
      ↩︎ Retraining when new labels arrive, without polling
    6. 6.
      “When you create an alert, the alert is suspended by default.”
      ↩︎ Turning the policy on: alerts and tasks start suspended
      “Instead, the failure occurs when the alert executes.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy
      “When you create an alert, the alert is suspended by default.”
      ↩︎ Exam trap 1
    7. 7.
      “ACTION_FAILED: The condition was evaluated successfully, but the execution of the action failed.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy
      “CONDITION_FAILED: The evaluation of the condition failed.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy
    8. 8.
      “SHOW ALERTS reports the alert in the SUSPENDED state with was_auto_suspended = TRUE.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy
      “The counter resets to 0 when the alert run succeeds, or when the alert is manually suspended or resumed.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy
      “SHOW ALERTS reports the alert in the SUSPENDED state with was_auto_suspended = TRUE.”
      ↩︎ Exam trap 4
      “alerts are automatically suspended after the specified number of consecutive alert runs that either fail or time out”
      ↩︎ Checkpoint
    9. 9.
      “The maximum size for a serverless task run is equivalent to an XXLARGE warehouse.”
      ↩︎ Troubleshooting pipeline failures that surface at the retraining policy

    Continue to part 2 of 2

    Troubleshooting failed and skipped retraining pipeline runs

    Spotted a mistake, or was something unclear? Tell us.