CertSafari
    Snowflake SnowPro Advanced: MLOps Engineer (MLA-B01)· Lessons

    Domain 4 · Lesson 14/17

    Troubleshooting failed and skipped retraining pipeline runs

    Implement retraining and troubleshooting.

    11 min read
    7.34% of exam
    6 sources
    Published 21 Oct 2026
    Docs as of 18 Oct 2026

    What you will be able to do

    • Use TASK_HISTORY, task state and owner privileges to tell a task that never ran from one whose SQL failed
    • Recognise warehouse undersizing from canceled runs, timeouts and skipped graph runs, and choose between resizing and raising USER_TASK_TIMEOUT_MS
    • Diagnose container jobs stuck in pending as compute pool resource starvation
    • Trace failures to stale streams, suspended model monitors and failed upstream dynamic tables
    • Rerun a failed retraining graph from the failed task and configure automatic retries

    1.Start from the task history, not the model

    When retraining doesn't happen, the model is rarely the cause. Usually the pipeline that should have retrained it failed, was skipped, or never started. Each of those has a different fix, so begin by working out which one happened. Query the TASK_HISTORY table function for the root task and each child. A task can run successfully while the SQL in its body fails, so read the error code and message as well as the status, and compare the scheduled and completed times.

    If there is no history for a task, look at its state and its owner. Tasks are created suspended, so check that every task in the graph shows RESUMED in DESCRIBE TASK or SHOW TASKS. The owning role also needs a set of privileges, listed below. Revoking EXECUTE TASK from a role stops every later run of every task that role owns. That makes it the first thing to check when a whole retraining pipeline stops at once.

    Privileges the task owner needs before any retraining task can run
    ObjectPrivilege
    AccountEXECUTE TASK
    DatabaseUSAGE
    SchemaUSAGE
    TaskOWNERSHIP
    WarehouseUSAGE

    Checkpoint 1 of 6· Check yourself

    A retraining pipeline worked for weeks. Then, from one moment on, TASK_HISTORY shows no new runs for any task the pipeline's owning role owns. That includes the root task and an unrelated nightly scheduled task. SHOW TASKS still lists every task as RESUMED. What should you check first?

    Sources1

    2.Warehouse capacity: timeouts, cancellations and skipped runs

    Training steps are heavy, and the warehouse is where they first run out of room. A task run has a 60-minute default limit. When TASK_HISTORY shows a training task that was canceled, or that overran its scheduled window, the usual cause is an undersized warehouse. Fix those symptoms before tuning anything else. Sizing and cost trade-offs are covered in Warehouse sizing and billing for ML workloads.

    There are two ways to fix it. A bigger warehouse makes the run finish inside the limit. A higher USER_TASK_TIMEOUT_MS on the task lets the run take longer. If the parameter check below returns no record, the task is using the default of 3,600,000 ms. Neither fix helps if the statement doesn't parallelize. In that case, rewrite the SQL. Also look at parallel branches: sibling child tasks run at the same time, and if they share a user-managed warehouse, it has to be sized for all of them together.

    Check whether a task has a custom timeoutsql
    SHOW PARAMETERS LIKE 'USER_TASK_TIMEOUT_MS' IN TASK <task_name>;

    Skipped runs are a capacity symptom. By default only one instance of a graph runs at a time (OVERLAP_POLICY = NO_OVERLAP). If a slow warehouse pushes total runtime past the root task's interval, the next run is dropped, and a skipped root task doesn't start the finalizer either. ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP allow concurrent instances, but they are only safe when overlapping runs don't write incorrect or duplicate data. For a graph that registers model versions, the better fix is usually more compute, serverless tasks, or a longer interval.

    Checkpoint 2 of 6· Check yourself

    After a training task was canceled at the 60-minute limit, an engineer moved it to a larger warehouse. They also raised USER_TASK_TIMEOUT_MS. The query profile shows the training statement still runs mostly on one thread, and the run still overruns. What is the right next step?

    Sources12

    3.Container capacity: jobs stuck pending on the compute pool

    Containerized training runs into a different limit: nodes in a compute pool rather than warehouse size. Snowflake ML Jobs run as job services on a compute pool. A job that sits in pending, or a service whose status falls from running back to pending, is a sign that the pool's nodes are starved of resources. A multi-node retraining job that asks for more than the pool can supply stalls in exactly this way. Raising the timeout does nothing, because the job is waiting for capacity, not running slowly.

    Too many resource-hungry containers on one node is the usual cause. Set an explicit memory request in the service specification so the scheduler keeps them apart. Memory is currently the only resource you can request. On a node with 64 GB of RAM, requesting more than 32 GB guarantees that only one service or job service runs on that node. Check the per-node capacity of the pool's instance type in CREATE COMPUTE POOL. Shutting a pool down has its own trap: a suspended pool can stay in STOPPING because running services prevent it from stopping, and ALTER COMPUTE POOL <pool_name> STOP ALL clears them.

    Checkpoint 3 of 6· Check yourself

    Several resource-hungry ML Job services share one compute pool. One retraining job service keeps falling from running back to pending and never finishes. Which change addresses the cause?

    Sources3

    4.Data dependencies: stale streams, suspended monitors, failed upstream tables

    Dependency failures surface one step away from their cause. A frozen model monitor (suspended after five consecutive source-table refresh failures), a quiet stream and a skipped child task all trace upward, and training data built from dynamic tables extends that chain further. For those tables the rule is upstream first. A table in the UPSTREAM_FAILED state didn't fail itself: its state_message names the upstream table that did, and that table's own error, often a USER_ERROR such as a column renamed in a base table, is the one to fix. Once it refreshes successfully, the downstream tables recover on their next scheduled refresh without any action.

    Suspension spreads along the same chain. Like monitors, dynamic tables suspend after five consecutive scheduled refresh failures (SUSPENDED_DUE_TO_ERRORS), and downstream tables suspended as a cascade resume automatically when you resume the root-cause table. Tables you suspended by hand must be resumed one by one. Resuming restores schedulability but does not trigger a refresh, so retraining started right after a resume can still read stale rows. Staleness also builds slowly: a time_within_target_lag_ratio below 0.90 means the table is missing its freshness target most of the time.

    Checkpoint 4 of 6· Match them up

    Training data comes from a chain of dynamic tables and retraining has gone quiet. Match each observation to what it means or what to do first.

    Tap a term, then the definition that fits it.

    Sources1456

    5.Rerunning a retraining graph without starting over

    Once the cause is fixed, rerun from where the graph failed, not from the beginning. Rerunning from the start repeats data preparation and training that already succeeded. EXECUTE TASK … RETRY LAST retries the latest graph run starting at the last failed task, and successful children continue as their parents complete. To retry an older run, use EXECUTE TASK … RETRY GRAPH RUN GROUP with that run's GRAPH_RUN_GROUP_ID.

    For transient failures, let the graph recover on its own. TASK_AUTO_RETRY_ATTEMPTS on the root task retries the whole graph immediately when a child fails, instead of waiting for the next trigger. SUSPEND_TASK_AFTER_NUM_FAILURES sets how many consecutive failures the graph tolerates before it suspends itself, 10 by default. A suspended graph won't respond to the alert's EXECUTE TASK the way you expect until it is resumed. Use SYSTEM$TASK_DEPENDENTS_ENABLE on the root task to resume every task in one call.

    A root task that retries failed graph runs twice and suspends after three consecutive failuressql
    CREATE OR REPLACE TASK task_root
      SCHEDULE = '1 MINUTE'
      TASK_AUTO_RETRY_ATTEMPTS = 2   --  Failed task graph retries up to 2 times
      SUSPEND_TASK_AFTER_NUM_FAILURES = 3   --  Task graph suspends after 3 consecutive failures
      AS SELECT 1;

    Checkpoint 5 of 6· Check yourself

    Last night's retraining run finished data preparation and training. Then the evaluation task failed because a source table was missing. You have restored the table. How do you finish that run without repeating training?

    Checkpoint 6 of 6· Fill the gap

    Which parameter makes this graph retry right away when a child task fails, instead of waiting for the next trigger?

    CREATE OR REPLACE TASK task_root
      SCHEDULE = '1 MINUTE'
       ?  = 2   --  Failed task graph retries up to 2 times
      SUSPEND_TASK_AFTER_NUM_FAILURES = 3   --  Task graph suspends after 3 consecutive failures
      AS SELECT 1;

    Sources2

    Sources

    (6)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Query the TASK_HISTORY table function to verify the task did not run.”
      ↩︎ Start from the task history, not the model
      “It is possible that the task ran successfully but the SQL statement in the task definition failed.”
      ↩︎ Start from the task history, not the model
      “Revoking the EXECUTE TASK privilege on a role prevents all subsequent task runs from starting under that role.”
      ↩︎ Start from the task history, not the model
      “There is a 60 minute default limit on a single run of a task.”
      ↩︎ Warehouse capacity: timeouts, cancellations and skipped runs
      “If the task was canceled or exceeded the window scheduled for the task, the cause is often an undersized warehouse.”
      ↩︎ Warehouse capacity: timeouts, cancellations and skipped runs
      “verify that the specified stream contained change data capture (CDC) records when the task was last scheduled to run”
      ↩︎ Data dependencies: stale streams, suspended monitors, failed upstream tables
      “Note that neither increasing the warehouse size nor increasing the timeout limit might help if there are query parallelization issues.”
      ↩︎ Exam trap 2
      “neither increasing the warehouse size nor increasing the timeout limit might help if there are query parallelization issues”
      ↩︎ Checkpoint
    2. 2.
      “When tasks run in parallel on the same user-managed warehouse, the compute resources must be sized to handle the concurrent task runs.”
      ↩︎ Warehouse capacity: timeouts, cancellations and skipped runs
      “OVERLAP_POLICY = NO_OVERLAP (default): Executes tasks serially with no parallelism.”
      ↩︎ Warehouse capacity: timeouts, cancellations and skipped runs
      “when read/write SQL operations executed by overlapping runs of a task graph do not produce incorrect or duplicate data”
      ↩︎ Warehouse capacity: timeouts, cancellations and skipped runs
      “to attempt to run the task graph from the last failed task”
      ↩︎ Rerunning a retraining graph without starting over
      “When a child task fails, the entire task graph is immediately retried, up to the number of times specified.”
      ↩︎ Rerunning a retraining graph without starting over
      “By default, a task graph is suspended after 10 consecutive failures.”
      ↩︎ Rerunning a retraining graph without starting over
      “at least one run of the task graph is skipped”
      ↩︎ Prediction
      “To retry the latest graph run, use EXECUTE TASK … RETRY LAST to attempt to run the task graph from the last failed task.”
      ↩︎ Checkpoint
    3. 3.
      “This could be an indication of resource starvation on compute pool nodes.”
      ↩︎ Container capacity: jobs stuck pending on the compute pool
      “would guarantee that only one service or job service can be running on a node at a time”
      ↩︎ Container capacity: jobs stuck pending on the compute pool
      “In the current implementation, you can only specify memory requests.”
      ↩︎ Container capacity: jobs stuck pending on the compute pool
      “Running services will prevent the compute pool from stopping.”
      ↩︎ Container capacity: jobs stuck pending on the compute pool
      “If your containers are resource-intensive (CPU/memory), you should explicitly specify resource requests in the service specification to prevent too many services”
      ↩︎ Checkpoint
    4. 4.
      “Model monitors automatically suspend refreshes when they encounter five consecutive refresh failures related to the source tables.”
      ↩︎ Data dependencies: stale streams, suspended monitors, failed upstream tables
      “Model monitors automatically suspend refreshes when they encounter five consecutive refresh failures related to the source tables.”
      ↩︎ Exam trap 1
    5. 5.
      “the downstream UPSTREAM_FAILED tables recover automatically on their next scheduled refresh”
      ↩︎ Data dependencies: stale streams, suspended monitors, failed upstream tables
      “Resuming restores schedulability but does not trigger an immediate refresh.”
      ↩︎ Data dependencies: stale streams, suspended monitors, failed upstream tables

    Ready to test yourself?

    Practise the 26 questions on this subdomain.

    Spotted a mistake, or was something unclear? Tell us.