CertSafari
    Snowflake SnowPro Advanced: MLOps Engineer (MLA-B01)· Lessons

    Domain 2 · Lesson 6/17

    Snowflake ML Jobs and custom runtime images

    Manage infrastructure for ML.

    13 min read
    8% of exam
    3 sources
    Published 21 Oct 2026
    Docs as of 18 Oct 2026

    What you will be able to do

    • Submit an ML Job with @remote, submit_file, submit_directory or submit_from_stage, and pin its runtime
    • Monitor an ML Job and get its result and logs
    • Run a multi-node ML Job with target_instances and min_instances
    • Build, validate, register and use a Custom Runtime Environment for ML Jobs

    1.Submitting work as an ML Job

    Snowflake ML Jobs run ML code in Container Runtime on a compute pool, started from whatever environment you already work in, such as VS Code, Jupyter or an Airflow task. You don't need a Snowflake notebook. You need snowflake-ml-python 1.26.0 or later and a Snowpark Session. Calling list_jobs() before you have a session fails.

    There are two ways to submit a job. Function dispatch uses the @remote decorator: it serializes the function and its dependencies, uploads them to a stage, and runs the function in Container Runtime. File dispatch submits whole scripts or projects. It suits code with several modules or command-line arguments, and existing projects that weren't written for Snowflake.

    ML Job submission methods
    MethodPayloadUse when
    @remoteOne Python functionOffloading a function from your IDE
    submit_fileA single Python fileA script that takes command-line arguments
    submit_directoryA local project directory plus an entrypointA project that spans several files and resources
    submit_from_stageA project already on a Snowflake stageProduction code kept on a stage rather than a laptop
    Submitting a single file with arguments to a compute poolpython
    from snowflake.ml.jobs import submit_file
    
    # Run a single file
    job1 = submit_file(
      "train.py",
      "MY_COMPUTE_POOL",
      stage_name="payload_stage",
      args=["--data-table", "my_training_data"],
      session=session,
    )

    All four methods accept runtime_environment. If you leave it out, the job runs on the latest Container Runtime available on the pool. That's convenient, but the environment changes under you as new versions come out. To pin a version, pass the full version string, such as runtime_environment="2.3.0". To use a custom image, pass cre@<name>. The file methods also take imports for extra code to ship with the job, but only single Python files can be loaded from a stage.

    A Snowpark Session is available inside the running job, so the payload can read tables and stages. Session.builder.getOrCreate() works for every payload type. Taking a session parameter in the function signature and having it injected works only for function dispatch.

    Checkpoint 1 of 6· Fill the gap

    A training script reads the name of its input table from the command line. Which keyword argument completes this submission so the script gets that value?

    from snowflake.ml.jobs import submit_file
    
    # Run a single file
    job1 = submit_file(
      "train.py",
      "MY_COMPUTE_POOL",
      stage_name="payload_stage",
       ? =["--data-table", "my_training_data"],
      session=session,
    )

    Checkpoint 2 of 6· Check yourself

    A nightly ML Job must keep running in the same environment until the team decides to upgrade. What should the submission do?

    Sources1

    2.Monitoring jobs and getting results

    Every submission method returns an MLJob object, and you manage the job through it. You can also get the object back later: list_jobs() lists jobs and get_job() fetches one by ID. A job's status is one of PENDING, RUNNING, FAILED or DONE.

    Listing, retrieving and waiting on ML Jobspython
    from snowflake.ml.jobs import MLJob, get_job, list_jobs
    
    # List all jobs
    jobs = list_jobs()
    
    # Retrieve an existing job based on ID
    job = get_job("<job_id>")  # job is an MLJob instance
    
    # Basic job information
    print(f"Job ID: {job.id}")
    print(f"Status: {job.status}")  # PENDING, RUNNING, FAILED, DONE
    
    # Wait for completion
    job.wait()

    result() blocks the calling thread until the job reaches a terminal state. Then it returns the payload's return value, or raises an exception if the run failed. How the payload provides that value depends on how you submitted it. A @remote function just returns it, for example a fitted model. A file-based job has no function to return from, so its script assigns the value to the special __return__ variable, usually __return__ = main() under the if __name__ == "__main__": guard. If the payload sets no return value, a successful run gives None.

    Logs come from the same object: job.get_logs() returns them and job.show_logs() prints them. A distributed job has a separate log for each node.

    Checkpoint 3 of 6· Check yourself

    Your team submits train.py with submit_file. The script fits a model inside main() and ends with if __name__ == "__main__": main(). The job's status shows DONE, but job.result() gives None. What is the fix?

    Sources1

    3.Multi-node ML Jobs

    A multi-node ML Job spreads one job across several nodes of a pool. One node is the head node, which coordinates. The rest are worker nodes, and all of them run the workload together as one logical job. A single-node job has only a head node. Adding nodes speeds up work only if the code is written for distribution, using Distributed Modeling Classes such as XGBEstimator, or Ray tasks. An ordinary single-process script still runs on one node.

    You request width with target_instances, using the same submission methods as a single-node job. The pool's MAX_NODES must be at least that number. Before running the payload, the job waits until the target number of nodes is ready, because workers can start seconds or minutes apart when the pool is short of capacity. If the nodes don't arrive within the timeout, the job fails. So a job asking for 8 nodes on a pool with MAX_NODES = 4 never gets them and fails with an error. It doesn't quietly train on 4.

    Preferring five nodes but starting once three are availablepython
    @remote(
        "MY_COMPUTE_POOL",
        stage_name="payload_stage",
        session=session,
        target_instances=5,  # Prefer 5 nodes
        min_instances=3      # But start with at least 3 nodes
    )

    min_instances lets a job start with fewer nodes. The payload starts as soon as that minimum is available, even if it's below target_instances. That cuts waiting on a busy pool and lets a workflow cope with whatever capacity it gets. The payload then has to adapt. A common pattern counts ray.nodes() and passes the count to XGBScalingConfig(num_workers=...).

    To debug, read a specific node's log with job.get_logs(instance_id=1). Calling it without an ID returns the head node's log. Instance 0 is not necessarily the head node, so don't assume instance_id=0 gives you the coordinator.

    Checkpoint 4 of 6· Check yourself

    An ML Job is submitted with target_instances=8 to a compute pool created with MAX_NODES = 4. No min_instances is set. What happens?

    Checkpoint 5 of 6· Fill the gap

    On a busy pool, a distributed training job should not sit waiting for all five nodes. It should start once three are ready. Which parameter completes the decorator?

    @remote(
        "MY_COMPUTE_POOL",
        stage_name="payload_stage",
        session=session,
        target_instances=5,  # Prefer 5 nodes
         ? =3      # But start with at least 3 nodes
    )

    Sources2

    4.Custom runtime images for ML Jobs

    A Custom Runtime Environment (CRE) is a container image registered with Snowflake so that ML Jobs and Notebooks run in it instead of the stock Container Runtime image. The path from a built image to a running job has three gates: local validation, server-side registration, and a digest check at submission.

    Build for linux/amd64 from an official Snowflake ML runtime base image, which comes per hardware type (CPU or GPU) and Python version. Then check the image locally with snow custom-image validate. This check is advisory, but it catches problems before Snowflake rejects the image at registration. To pass:

    • the entrypoint must be exactly /usr/local/bin/entrypoint.sh
    • the environment must set DASHBOARD_PORT=12003
    • the mandatory packages must be installed, including snowflake-ml-python, ray and ipykernel
    • the dependency tree must resolve with no conflicts

    Add --scan-vulnerabilities to also scan for CVEs. That scan isn't required to pass.

    Registering the pushed image as a Custom Runtime Environmentsql
    CREATE CUSTOM RUNTIME ENVIRONMENT my_cre
        IMAGE_PATH = '/<database>/<schema>/<image_repo>/my-custom-image:v1'
        BASE_IMAGE_TYPE = CPU;

    After you tag the image and push it to a Snowflake image repository, register it with CREATE CUSTOM RUNTIME ENVIRONMENT. Snowflake runs a server-side check that the image actually executes, and creates the CRE only if that check passes. The CRE is then in the RESOLVED state. To use it, a job passes runtime_environment="cre@my_cre". That needs snowflake-ml-python 1.37.0 or later, a role with USAGE on the CRE plus access to the image repository, and a CRE that is not DEPRECATED. Registered CREs can also be chosen for notebooks in Workspaces, where notebook runtimes are set up.

    A CRE is tied to one exact image, not to a tag. When a job is submitted, Snowflake checks the image's SHA digest. If someone pushed a changed image to the same path after registration, the submission is rejected. To ship a fixed image, create the CRE again.

    Checkpoint 6 of 6· Put it in order

    Put these steps in order to run an ML Job in your own image, which has libraries the stock Container Runtime lacks.

    1. 1.Submit the job with runtime_environment="cre@my_cre"
    2. 2.Check the image locally with snow custom-image validate
    3. 3.Tag the image with the Snowflake image repository path and push it
    4. 4.Write a Dockerfile FROM an official Snowflake ML runtime base image and build it for linux/amd64
    5. 5.Register the image with CREATE CUSTOM RUNTIME ENVIRONMENT, which runs a server-side check

    Sources3

    Practise it for real

    Turn a pinned-library Docker image into a Custom Runtime Environment and run an ML Job in it

    1. 1.Run snow spcs image-registry login, then docker pull the CPU base image container_runtime/cpu_x86_64:2.7.2 from your registry URL.

      Why: Every custom image must derive from an official Snowflake ML runtime base image, and Docker has to be logged in to the Snowflake registry to pull it.

      You should see: The base image appears in docker images.

    2. 2.Write a Dockerfile FROM that base image that installs xgboost==2.0.3 and scikit-learn==1.3.2, then run docker build --platform linux/amd64 -t my-custom-image:v1 .

      Why: Snowflake runs linux/amd64 images, and pinning versions makes the environment reproducible.

      You should see: A local image tagged my-custom-image:v1.

    3. 3.Run snow custom-image validate my-custom-image:v1 --scan-vulnerabilities.

      Why: Validating locally catches a wrong entrypoint, a missing DASHBOARD_PORT, missing mandatory packages or dependency conflicts before registration fails on the server.

      You should see: A validation report with no blocking errors. CVE findings are advisory.

    4. 4.Tag the image with <registry_url>/<database>/<schema>/<image_repo>/my-custom-image:v1, docker push it, then run CREATE CUSTOM RUNTIME ENVIRONMENT my_cre with IMAGE_PATH and BASE_IMAGE_TYPE = CPU.

      Why: Registration runs the server-side check and ties the CRE to this exact image digest.

      You should see: The CRE is created in the RESOLVED state.

    5. 5.Decorate a function with @jobs.remote("MY_COMPUTE_POOL", stage_name="payload_stage", runtime_environment="cre@my_cre"), call it, then call job.result().

      Why: This shows the job ran in your image and not in the default Container Runtime.

      You should see: The job reaches DONE and result() returns the function's value.

    Stuck? Get a nudge

    If submission is rejected after you rebuilt and re-pushed the same tag, the digest check is working as designed. Create the CRE again.

    Sources

    (3)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Snowflake ML Jobs are available in snowflake-ml-python version 1.26.0 and later.”
      ↩︎ Submitting work as an ML Job
      “Snowflake automatically uses the latest available version of the Snowflake Container Runtime on your compute pool.”
      ↩︎ Submitting work as an ML Job
      “You can only load single Python files from a stage.”
      ↩︎ Submitting work as an ML Job
      “The API blocks the calling thread until the job reaches a terminal state”
      ↩︎ Monitoring jobs and getting results
      “For file-based jobs, use the special __return__ variable to specify the return value.”
      ↩︎ Monitoring jobs and getting results
      “For file-based jobs, use the special __return__ variable to specify the return value.”
      ↩︎ Exam trap 3
      “To specify a container runtime for your ML Job, use the runtime_environment keyword with the full Container Runtime version string.”
      ↩︎ Checkpoint
      “if execution failed, raises an exception.”
      ↩︎ Prediction
    2. 2.
      “For multi-node jobs, your code needs to be designed for distributed processing using Distributed Modeling Classes or Ray.”
      ↩︎ Multi-node ML Jobs
      “The job fails with an error if the expected nodes aren’t available within the timeout period.”
      ↩︎ Multi-node ML Jobs
      “the job payload is executed as soon as the minimum number of nodes becomes available”
      ↩︎ Multi-node ML Jobs
      “The job fails with an error if the expected nodes aren’t available within the timeout period.”
      ↩︎ Exam trap 1
    3. 3.
      “All custom images must be derived from an official Snowflake ML runtime base image.”
      ↩︎ Custom runtime images for ML Jobs
      “During this step, Snowflake performs a server-side validation check to confirm the image executes correctly.”
      ↩︎ Custom runtime images for ML Jobs
      “The active role must hold the USAGE privilege on the target Custom Runtime Environment, along with access to the underlying image repository.”
      ↩︎ Custom runtime images for ML Jobs
      “If the underlying image in the repository was modified after the CRE was registered, the job submission is rejected to ensure reproducibility.”
      ↩︎ Exam trap 2
      “Before pushing the image to Snowflake, validate it locally to ensure it meets runtime requirements.”
      ↩︎ Checkpoint

    Ready to test yourself?

    Practise the 29 questions on this subdomain.

    Spotted a mistake, or was something unclear? Tell us.