CertSafari
    Snowflake SnowPro Advanced: MLOps Engineer (MLA-B01)· Lessons

    Domain 4 · Lesson 12/17

    Orchestrating ML pipelines with Snowflake task graphs

    Orchestrate end-to-end ML workflows.

    23 min read
    7.33% of exam
    7 sources
    Published 20 Oct 2026
    Docs as of 18 Oct 2026

    What you will be able to do

    • Restructure notebook code into modular, parameterized pipeline steps with a local entrypoint script
    • Run each pipeline step as an ML Job whose result and runtime are fixed so the steps can be chained
    • Build a task graph in which a failed validation step stops training and deployment
    • Pass values between tasks, trigger a graph on new data, and choose an overlap policy and finalizer behaviour
    • Resume, run, retry and monitor a task graph, and start runs by hand from Snowflake CLI, SnowSQL and the Python SDK

    Key concept

    Task graph (DAG) — A root task that decides when the pipeline runs, plus child tasks linked to their parents with AFTER. A child starts only after every parent has succeeded, so the order of the graph is also its quality gate.

    1.From experiment to an automated ML pipeline

    An ML workflow has four stages: data preparation, feature engineering, model development and deployment. Each one starts life as an interactive experiment. Once a model proves its value, the work moves to operationalization, where the pipeline is hardened and automated so that every change to code, data or model is built, tested and deployed the same way each time. In Snowflake, the orchestration layer for this is a task graph: the pipeline turned into a directed acyclic graph that runs on a schedule or when an event triggers it.

    A task graph has one root task and any number of dependent child tasks. The root task decides when the graph runs. Each child names its parents with CREATE TASK ... AFTER, and dependencies always run from start to finish, with no loops. Children that share a parent run in parallel. A child with several parents waits until all of them have finished successfully. One graph can hold up to 1000 tasks, and a single task can have at most 100 parents and 100 children.

    A root task scheduled every minute, two parallel children, and a third child that waits for bothsql
    CREATE TASK task_root
      SCHEDULE = '1 MINUTE'
      AS SELECT 1;
    
    CREATE TASK task_a
      AFTER task_root
      AS SELECT 1;
    
    CREATE TASK task_b
      AFTER task_root
      AS SELECT 1;
    
    CREATE TASK task_c
      AFTER task_a, task_b
      AS SELECT 1;

    Every task in a graph must have the same owner and live in the same database and schema. Ownership matters more than it looks. Granting ownership of one task to a different role cuts that task's links to its parents and children. Moving the whole graph to a new owner keeps the links intact, either by GRANT OWNERSHIP on all tasks in the schema or by dropping the owning role. If parallel children share a user-managed warehouse, size that warehouse for the concurrent runs.

    Checkpoint 1 of 11· Fill the gap

    This task graph needs task_a and task_b to run in parallel straight after the root, and task_c to wait for both of them. Which task name fills the blank?

    CREATE TASK task_root
      SCHEDULE = '1 MINUTE'
      AS SELECT 1;
    
    CREATE TASK task_a
      AFTER task_root
      AS SELECT 1;
    
    CREATE TASK task_b
      AFTER  ? 
      AS SELECT 1;
    
    CREATE TASK task_c
      AFTER task_a, task_b
      AS SELECT 1;

    Sources12

    2.Restructuring notebook code into modular steps

    A notebook cannot become a pipeline as it stands. Each major step needs to be its own function with clear inputs and outputs: data preparation, feature engineering, training and evaluation. Hard-coded values become parameters. Table names, stages and hyperparameters come from configuration, so the same code runs in dev and in prod. Keeping steps modular from the first interactive session, with one cell or function per stage, makes this refactor small. Deciding where that code should live, whether in Workspaces, stored procedures or Python files, is covered in When to develop in Workspaces vs stored procedures, SQL or Python files. For a pipeline, the cells end up as modules under a package such as src/ml_pipeline/, with data/, features/, models/ and inference/ packages.

    Write an entrypoint script as well, one that runs the whole pipeline locally. It is the version you debug, and the graph definition (dag.py) sits next to it in scripts/. The entrypoint reads --config and --env, loads the matching configuration, and calls the steps in order. Validation sits between loading and feature creation, so bad data never reaches training.

    Entrypoint that chains parameterized steps: load, validate, featurize, train, evaluate, registerpython
    def main():
        parser = argparse.ArgumentParser()
        parser.add_argument("--config", required=True, help="Config file path")
        parser.add_argument("--env", default="dev", help="Environment (dev/prod)")
        args = parser.parse_args()
    
        # Load configuration
        config = load_config(args.config, args.env)
    
        # Execute pipeline stages
        raw_data = load_raw_data(config.data.source_table)
        validate_data_quality(raw_data, config.data.quality_checks)
        features = create_features(raw_data, config.features.transformations)
        model = train_model(features, config.model.hyperparameters)
        metrics = evaluate_model(model, features, config.model.eval_metrics)
        register_model(model, metrics, config.model.registry_name)

    Checkpoint 2 of 11· Check yourself

    A data scientist's training notebook has the dev source table name and the XGBoost hyperparameters typed straight into its cells. The team wants the same code to run as a pipeline in both dev and prod. What should they change before building the task graph?

    Sources1

    3.Running each step as an ML Job

    Heavy steps such as training are dispatched to Snowflake compute as ML Jobs, through a @remote function or submit_file / submit_directory / submit_from_stage. The mechanics are covered in Snowflake ML Jobs. Here the question is what a pipeline needs from a job for it to be chainable, and there are three things.

    A result to collect. A @remote function returns its value directly. A file-based job sets the special __return__ variable instead. The pipeline then calls MLJob.result(), which blocks until the job reaches a terminal state and then either returns the value or raises an exception if the job failed. That exception matters for orchestration: a task that calls result() on a failed training job fails too, and the steps after it do not run.

    A pinned runtime. If you leave out runtime_environment, the job runs on the latest Container Runtime available on the compute pool. That runtime can change between Tuesday's run and Wednesday's. Pinning a full version string such as "2.3.0", or a custom image with "cre@<name>", keeps unattended runs reproducible.

    Imports in the right place. Most notebook code runs unchanged in an ML Job. The exception is the distributed ML APIs that exist only inside the Container Runtime: import them inside the job payload, not at the top of the orchestrating script, or the import fails.

    Checkpoint 3 of 11· Check yourself

    In a task graph, a train task submits training as a file-based ML Job, and an evaluate task runs AFTER it. One night the training job fails on the compute pool. What stops the evaluate task from running on a model that was never trained?

    Sources31

    4.Gating training on a validation step

    The typical ML graph chains VALIDATE_DATA → TRAIN_MODEL → DEPLOY_MODEL. The validation step can run Data Metric Functions and expectations against the new data. In the graph, its job is to act as a gate: when a check fails, training and deployment must not run.

    Failure is the gate. A child task starts only once its predecessors have completed successfully, and by default one failed child marks the whole graph run as failed. So the validation procedure should raise an exception when a threshold is breached. Its task then fails, TRAIN_MODEL and DEPLOY_MODEL never start, and the failure is recorded in task history where anyone can see it.

    The alternatives break in specific ways. Returning the text FAILED produces a successful task, so the child starts anyway. A return value only affects control flow if the child's own body contains logic that reads it and acts on it. A separate validation root task does not work either, because each root task schedules its own graph and nothing connects the two. A check in the finalizer comes too late, since the finalizer runs after training has already finished.

    Use a deliberate branch when the outcome is a decision rather than an error. The Snowflake end-to-end example evaluates the trained model and picks the next task by name. If the model clears the quality threshold, the promotion task registers it and sets it as the default version, so callers of the model pick it up without changing their code (see the Model Registry). If it falls short, an alert task runs instead.

    A branch task in the Python task API returns the name of the next task to runpython
    def check_model_quality(session: Session) -> str:
        if metrics[config.metric_name] >= threshold:
            return "promote_model"  # High quality → promote
        else:
            return "send_alert"     # Low quality → alert

    Checkpoint 4 of 11· Fill the gap

    After evaluation, the pipeline should either promote the model or send an alert. The model clearing the bar is a decision, not an error. Which return type completes this branch task from the Python task API?

    def check_model_quality(session: Session) ->  ? :
        if metrics[config.metric_name] >= threshold:
            return "promote_model"  # High quality → promote
        else:
            return "send_alert"     # Low quality → alert

    Checkpoint 5 of 11· Exam question

    A nightly task graph chains VALIDATE_DATA, TRAIN_MODEL, and DEPLOY_MODEL. When the validation step finds a null rate above the agreed threshold, training and deployment must not run, and the failure must show up in task history. How should the validation step be built?

    Sources24

    5.Passing values between tasks and triggering on new data

    Tasks hand data to each other through return values. A task calls SYSTEM$SET_RETURN_VALUE('<string>'), and any task that names it in its AFTER clause reads the value with SYSTEM$GET_PREDECESSOR_RETURN_VALUE. This is how TRAIN_MODEL passes the version name it just created to DEPLOY_MODEL in the same graph run. The value is a string of at most 10 kB, so pass an identifier such as a version name or a table name, not the model itself. In the Python task API the same handoff goes through the TaskContext.

    The root task decides when the graph runs, either on a schedule or on an event. For retraining on newly arrived data, a triggered task is the better fit when the arrival time is unpredictable, for example when rows land continuously through Snowpipe. You create a stream on the source table, and the root task names that stream in a WHEN clause and has no SCHEDULE. A triggered task uses no compute until the stream has changes, so nothing is spent polling an empty table. Streams on tables, views, dynamic tables, Iceberg tables and data shares can trigger tasks. Hybrid tables and streams on external tables cannot.

    Choosing how the root task starts the graph
    Root task settingRuns whenFits
    SCHEDULE = '1 MINUTE' (or a longer interval)The interval elapses, whether or not there is new dataNightly or periodic retraining on a fixed cadence
    WHEN on a stream, no SCHEDULEThe stream records a changeRetraining only when new training rows arrive

    Checkpoint 6 of 11· Check yourself

    New labelled rows arrive in a training table through Snowpipe at unpredictable times, sometimes several times a day and sometimes not for a week. The team wants the retraining graph to run only when new rows have landed and to spend nothing in between. How should they set up the graph?

    Sources564

    6.Finalizer tasks and overlapping runs

    A finalizer is an optional last task that runs after every other task in the graph run has finished, whether they succeeded or failed. That makes it the place for cleanup, such as dropping temporary training tables, and for sending a success or failure notification. You attach it to the root task with FINALIZE, not AFTER.

    Attaching a finalizer to the root tasksql
    CREATE TASK task_finalizer
      FINALIZE = task_root
      AS SELECT 1;

    A finalizer has strict rules:

    • Each root task has at most one finalizer, and a finalizer belongs to exactly one root.
    • A finalizer cannot have child tasks.
    • It is scheduled only once no other task in the run is running or queued.
    • If the root task is skipped, for example because of an overlapping run, the finalizer is not started at all.

    Because the finalizer runs on failure as well as success, it is the wrong place for model promotion, which should happen only when the run succeeds. Put promotion on the success path of the graph instead.

    Overlap is the other run-level setting. By default only one instance of a graph runs at a time, and the root is scheduled again only after every task has finished. If a training run takes longer than the schedule interval, at least one scheduled run is skipped. The OVERLAP_POLICY parameter on the root task changes this behaviour.

    OVERLAP_POLICY values set on the root task
    ValueBehaviourWhen it is safe
    NO_OVERLAP (default)Runs serially. The next root run is scheduled only after all child tasks finishRetraining that writes models or tables a second run would clash with
    ALLOW_CHILD_OVERLAPA new graph instance starts while children of the previous one are still running. Root tasks never overlapChild steps can safely run alongside the previous run's children
    ALLOW_ALL_OVERLAPSeveral full instances, root included, run concurrentlyOverlapping reads and writes cannot produce incorrect or duplicate data

    When overlap would be unsafe, the fix is to make sure each run finishes before the next one is due. Lengthen the schedule interval, size the warehouse up, or use serverless compute.

    Checkpoint 7 of 11· Match them up

    A nightly retraining graph sometimes runs past its next scheduled start time. Match each OVERLAP_POLICY value on the root task to what happens when that occurs.

    Tap a term, then the definition that fits it.

    Checkpoint 8 of 11· Exam question

    Which statements about a finalizer task in a Snowflake task graph are true? (Select two.)(Select 2)

    0 of 2 selected

    Sources2

    7.Starting, running and retrying a task graph

    New tasks are created suspended, and resuming the root does not resume its children. There are two ways to enable a graph. You can run ALTER TASK ... RESUME on each child, including the finalizer, and then on the root. Or you can make one call: SELECT SYSTEM$TASK_DEPENDENTS_ENABLE('<root_task_name>'), which resumes the root and every dependent task. To list a graph's tasks, pass the root task to the TASK_DEPENDENTS table function.

    EXECUTE TASK on the root starts one run immediately, which is useful for testing a changed graph before it goes on a schedule. It does not change any task's state. Only children that are already resumed take part in the run. To change a task in a scheduled graph, suspend the root first. A run already in progress finishes, and future runs are cancelled until the root is resumed. Suspending a single child skips it: the graph continues as if that child had succeeded.

    Failures can be retried without starting over:

    • EXECUTE TASK ... RETRY LAST reruns the latest graph run from the task that failed, so a fixed TRAIN_MODEL runs again without repeating a VALIDATE_DATA that already passed.
    • EXECUTE TASK ... RETRY GRAPH RUN GROUP with a GRAPH_RUN_GROUP_ID retries an older run.
    • TASK_AUTO_RETRY_ATTEMPTS on the root retries the whole graph immediately when a child fails.
    • SUSPEND_TASK_AFTER_NUM_FAILURES sets how many consecutive failures suspend the graph. The default is 10.
    Root task with automatic retries and a suspension thresholdsql
    CREATE OR REPLACE TASK task_root
      SCHEDULE = '1 MINUTE'
      TASK_AUTO_RETRY_ATTEMPTS = 2   --  Failed task graph retries up to 2 times
      SUSPEND_TASK_AFTER_NUM_FAILURES = 3   --  Task graph suspends after 3 consecutive failures
      AS SELECT 1;

    To find out which step failed, open Snowsight under Monitoring > Task History and select the graph. The view shows each task's status, logs and dependencies for the latest run, and which branch a conditional step took.

    Checkpoint 9 of 11· Fill the gap

    When a child task such as TRAIN_MODEL fails during the night, the team wants the whole graph to retry straight away instead of waiting for the next scheduled run. They also want the graph to suspend after three failures in a row. Which root-task parameter fills the blank?

    CREATE OR REPLACE TASK task_root
      SCHEDULE = '1 MINUTE'
       ?  = 2   --  Failed task graph retries up to 2 times
      SUSPEND_TASK_AFTER_NUM_FAILURES = 3   --  Task graph suspends after 3 consecutive failures
      AS SELECT 1;

    Checkpoint 10 of 11· Exam question

    A team defined a graph of six tasks with CREATE TASK ... AFTER statements. They want every task in the graph enabled with one call after deployment. Which approach does that?

    Sources2

    8.Driving pipelines from Snowflake CLI, SnowSQL and the SDKs

    While you develop a pipeline, you start runs by hand from whatever client is nearest. ML Jobs can be submitted from any development environment, such as VS Code or Jupyter. They need a Snowpark Session, which is built from the connection in ~/.snowflake/config.toml. Tasks and task graphs can be defined in SQL, JavaScript, Python, Java, Scala or Snowflake Scripting. Wiring these commands into automated deployment is covered in Snowflake CLI for automation.

    Developer tools and the part of the pipeline each one drives
    ToolStatusTypical use in an ML pipeline
    Snowflake CLIPreferred client for new installs and new workRun SQL such as EXECUTE TASK or SYSTEM$TASK_DEPENDENTS_ENABLE, and manage stored procedures and stages
    SnowSQLLegacy client, supported in version 1.5.x through April 16, 2028. New features go only to Snowflake CLIRun the same SQL statements in existing scripts
    Python SDK (snowflake-ml-python, Snowpark)Required for ML Jobs, version 1.26.0 and laterSubmit ML Jobs and define the graph with the DAG context manager, >> dependencies, TaskContext and DAGTaskBranch

    The Python route works well for iterating. In Snowflake's end-to-end example, pipeline_local.py runs the pipeline without any orchestration, and the training ML Job shows up in the Snowsight Job UI. pipeline_dag.py then deploys the same steps as a task graph. Its --run-dag flag deploys the graph, runs it once, follows its progress and prints the final status. Its --schedule flag deploys the graph to run on a recurring interval.

    Deploying the example graph on recurring schedules from the command linebash
    python src/pipeline_dag.py --schedule 1d   # Daily execution
    python src/pipeline_dag.py --schedule 12h  # Every 12 hours
    python src/pipeline_dag.py --schedule 30m  # Every 30 minutes

    Checkpoint 11 of 11· Match them up

    Your team is automating a retraining pipeline and has to choose a tool for each job. Match each tool to the role it should play.

    Tap a term, then the definition that fits it.

    Sources723

    Practise it for real

    Deploy, run and schedule the Snowflake end-to-end ML task graph sample, then inspect it in Snowsight

    1. 1.From the e2e_task_graph folder, run python src/pipeline_local.py --no-register

      Why: Runs the modular pipeline without orchestration, so step failures show up before a graph is involved

      You should see: The pipeline completes, and the training ML Job appears in the Snowsight Job UI

    2. 2.Run python src/pipeline_dag.py --run-dag

      Why: Deploys the same steps as a task graph and runs it once

      You should see: The script reports execution progress and prints the final status

    3. 3.Open Monitoring > Task History in Snowsight and select the DAG

      Why: Shows which tasks ran, which branch the quality check chose, and the logs of any failure

      You should see: Each task's status for the latest run, with either the promotion or the alert branch taken

    4. 4.Run python src/pipeline_dag.py --schedule 1d

      Why: Deploys the graph to run on a recurring schedule

      You should see: The graph is deployed to run daily

    Stuck? Get a nudge

    If training fails on node capacity, check that the compute pool's MAX_NODES is at least the job's target_instances.

    Sources

    (7)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Operationalize your ML pipeline into a Directed Acyclic Graph (DAG) and configure it to run on a schedule or by event based triggers.”
      ↩︎ From experiment to an automated ML pipeline
      “each major step (data preparation, feature engineering, model training, evaluation) becomes a separate function with clear inputs and outputs”
      ↩︎ Restructuring notebook code into modular steps
      “Parameterize all configuration values like table names and hyperparameters to enable cross-environment deployment.”
      ↩︎ Restructuring notebook code into modular steps
      “We recommend also authoring an entrypoint script which executes the end-to-end pipeline locally for debugging and future development.”
      ↩︎ Restructuring notebook code into modular steps
      “Certain distributed ML APIs are only available inside the Container Runtime, and attempting to import them outside the Container Runtime environment will fail.”
      ↩︎ Running each step as an ML Job
    2. 2.
      “The dependencies must run in a start-to-finish direction, with no loops.”
      ↩︎ From experiment to an automated ML pipeline
      “All tasks in a task graph must have the same task owner and be stored in the same database and schema.”
      ↩︎ From experiment to an automated ML pipeline
      “By default, if a child task fails, the entire task graph is considered to have failed.”
      ↩︎ Gating training on a validation step
      “When a task has multiple parents, the task waits for all preceding tasks to successfully complete before starting.”
      ↩︎ Gating training on a validation step
      “Each root task can have only one finalizer task, and a finalizer task can be associated with only one root task.”
      ↩︎ Finalizer tasks and overlapping runs
      “A finalizer task cannot have any child tasks.”
      ↩︎ Finalizer tasks and overlapping runs
      “A finalizer task is scheduled only when no other tasks are running or queued in the current task graph run.”
      ↩︎ Finalizer tasks and overlapping runs
      “OVERLAP_POLICY = NO_OVERLAP (default): Executes tasks serially with no parallelism.”
      ↩︎ Finalizer tasks and overlapping runs
      “Resume all of the tasks in a task graph at once using SYSTEM$TASK_DEPENDENTS_ENABLE ( <root_task_name> ) on the root task.”
      ↩︎ Starting, running and retrying a task graph
      “To retry the latest graph run, use EXECUTE TASK … RETRY LAST to attempt to run the task graph from the last failed task.”
      ↩︎ Starting, running and retrying a task graph
      “By default, a task graph is suspended after 10 consecutive failures.”
      ↩︎ Starting, running and retrying a task graph
      “When you suspend a child task, the task graph continues to run as though the child task had succeeded.”
      ↩︎ Starting, running and retrying a task graph
      “You can create tasks and task graphs using supported languages and tools like SQL, JavaScript, Python, Java, Scala, or Snowflake Scripting.”
      ↩︎ Driving pipelines from Snowflake CLI, SnowSQL and the SDKs
      “When a task has multiple parents, the task waits for all preceding tasks to successfully complete before starting.”
      ↩︎ Key concept
      “specifying logic-based operations in the task body using runtime values, graph level configuration, and return values of parent tasks”
      ↩︎ Exam trap 1
      “When the root task of a task graph is skipped (for example, because of overlap task graph runs), the finalizer task won’t be started.”
      ↩︎ Exam trap 2
      “When you run the root task, all resumed child tasks in the task graph are executed in the order defined by the task graph.”
      ↩︎ Exam trap 3
      “at least one run of the task graph is skipped”
      ↩︎ Exam trap 4
      “Task graphs can run on a recurring schedule, or they can be triggered by an event.”
      ↩︎ Checkpoint
      “OVERLAP_POLICY = ALLOW_ALL_OVERLAP: Allows unlimited true parallelism. Snowflake can run multiple instances of the entire task graph, including the root task, concurrently.”
      ↩︎ Checkpoint
    3. 3.
      “returns the payload’s return value or, if execution failed, raises an exception”
      ↩︎ Running each step as an ML Job
      “For file-based jobs, use the special __return__ variable to specify the return value.”
      ↩︎ Running each step as an ML Job
      “Snowflake automatically uses the latest available version of the Snowflake Container Runtime on your compute pool”
      ↩︎ Running each step as an ML Job
      “You can run them from any development environment.”
      ↩︎ Driving pipelines from Snowflake CLI, SnowSQL and the SDKs
      “then returns the payload’s return value or, if execution failed, raises an exception.”
      ↩︎ Checkpoint
    4. 4.
      “Branching Logic: Using DAGTaskBranch for conditional execution paths”
      ↩︎ Gating training on a validation step
      “Task Context: Passing values between Tasks in the DAG via the TaskContext”
      ↩︎ Passing values between tasks and triggering on new data
    5. 5.
      “In a task graph, a task can call this function to set a return value.”
      ↩︎ Passing values between tasks and triggering on new data
      “The string size must be <= 10 kB (when encoded in UTF8).”
      ↩︎ Passing values between tasks and triggering on new data
    6. 6.
      “Define the target stream using the WHEN clause. (Do not include the SCHEDULE parameter.)”
      ↩︎ Passing values between tasks and triggering on new data
      “Triggered tasks don’t use compute resources until the event is triggered.”
      ↩︎ Passing values between tasks and triggering on new data
    7. 7.
      “Snowflake CLI is the preferred command-line client for new installs and new work.”
      ↩︎ Driving pipelines from Snowflake CLI, SnowSQL and the SDKs
      “Snowflake supports SnowSQL 1.5.x through April 16, 2028.”
      ↩︎ Driving pipelines from Snowflake CLI, SnowSQL and the SDKs

    Ready to test yourself?

    Practise the 26 questions on this subdomain.

    Spotted a mistake, or was something unclear? Tell us.