What you will be able to do
- Submit an ML Job with @remote, submit_file, submit_directory or submit_from_stage, and pin its runtime
- Monitor an ML Job and get its result and logs
- Run a multi-node ML Job with target_instances and min_instances
- Build, validate, register and use a Custom Runtime Environment for ML Jobs
1.Submitting work as an ML Job
Snowflake ML Jobs run ML code in Container Runtime on a compute pool, started from whatever environment you already work in, such as VS Code, Jupyter or an Airflow task. You don't need a Snowflake notebook. You need snowflake-ml-python 1.26.0 or later and a Snowpark Session. Calling list_jobs() before you have a session fails.
There are two ways to submit a job. Function dispatch uses the @remote decorator: it serializes the function and its dependencies, uploads them to a stage, and runs the function in Container Runtime. File dispatch submits whole scripts or projects. It suits code with several modules or command-line arguments, and existing projects that weren't written for Snowflake.
| Method | Payload | Use when |
|---|---|---|
| @remote | One Python function | Offloading a function from your IDE |
| submit_file | A single Python file | A script that takes command-line arguments |
| submit_directory | A local project directory plus an entrypoint | A project that spans several files and resources |
| submit_from_stage | A project already on a Snowflake stage | Production code kept on a stage rather than a laptop |
from snowflake.ml.jobs import submit_file
# Run a single file
job1 = submit_file(
"train.py",
"MY_COMPUTE_POOL",
stage_name="payload_stage",
args=["--data-table", "my_training_data"],
session=session,
)All four methods accept runtime_environment. If you leave it out, the job runs on the latest Container Runtime available on the pool. That's convenient, but the environment changes under you as new versions come out. To pin a version, pass the full version string, such as runtime_environment="2.3.0". To use a custom image, pass cre@<name>. The file methods also take imports for extra code to ship with the job, but only single Python files can be loaded from a stage.
A Snowpark Session is available inside the running job, so the payload can read tables and stages. Session.builder.getOrCreate() works for every payload type. Taking a session parameter in the function signature and having it injected works only for function dispatch.
Checkpoint 1 of 6· Fill the gap
A training script reads the name of its input table from the command line. Which keyword argument completes this submission so the script gets that value?
from snowflake.ml.jobs import submit_file
# Run a single file
job1 = submit_file(
"train.py",
"MY_COMPUTE_POOL",
stage_name="payload_stage",
? =["--data-table", "my_training_data"],
session=session,
)File dispatch passes command-line arguments to the script through args. imports sends extra code along with the job. entrypoint names the script to start when you submit a directory or stage project. runtime_environment chooses the container runtime.
Checkpoint 2 of 6· Check yourself
A nightly ML Job must keep running in the same environment until the team decides to upgrade. What should the submission do?
If you leave out runtime_environment, the job uses the latest Container Runtime on the pool, so the environment changes when new versions come out. Passing the full version string pins it.
“To specify a container runtime for your ML Job, use the runtime_environment keyword with the full Container Runtime version string.”Source: docs.snowflake.com
Sources1
2.Monitoring jobs and getting results
Every submission method returns an MLJob object, and you manage the job through it. You can also get the object back later: list_jobs() lists jobs and get_job() fetches one by ID. A job's status is one of PENDING, RUNNING, FAILED or DONE.
from snowflake.ml.jobs import MLJob, get_job, list_jobs
# List all jobs
jobs = list_jobs()
# Retrieve an existing job based on ID
job = get_job("<job_id>") # job is an MLJob instance
# Basic job information
print(f"Job ID: {job.id}")
print(f"Status: {job.status}") # PENDING, RUNNING, FAILED, DONE
# Wait for completion
job.wait()result() blocks the calling thread until the job reaches a terminal state. Then it returns the payload's return value, or raises an exception if the run failed. How the payload provides that value depends on how you submitted it. A @remote function just returns it, for example a fitted model. A file-based job has no function to return from, so its script assigns the value to the special __return__ variable, usually __return__ = main() under the if __name__ == "__main__": guard. If the payload sets no return value, a successful run gives None.
Logs come from the same object: job.get_logs() returns them and job.show_logs() prints them. A distributed job has a separate log for each node.
Checkpoint 3 of 6· Check yourself
Your team submits train.py with submit_file. The script fits a model inside main() and ends with if __name__ == "__main__": main(). The job's status shows DONE, but job.result() gives None. What is the fix?
A file-based job has no function to return from, so result() only gets a value if the script assigns it to __return__. If the script sets no return value, a successful run gives None. result() already waits for a terminal state, and file dispatch can return values just as @remote can.
“For file-based jobs, use the special __return__ variable to specify the return value.”Source: docs.snowflake.com
Sources1
3.Multi-node ML Jobs
A multi-node ML Job spreads one job across several nodes of a pool. One node is the head node, which coordinates. The rest are worker nodes, and all of them run the workload together as one logical job. A single-node job has only a head node. Adding nodes speeds up work only if the code is written for distribution, using Distributed Modeling Classes such as XGBEstimator, or Ray tasks. An ordinary single-process script still runs on one node.
You request width with target_instances, using the same submission methods as a single-node job. The pool's MAX_NODES must be at least that number. Before running the payload, the job waits until the target number of nodes is ready, because workers can start seconds or minutes apart when the pool is short of capacity. If the nodes don't arrive within the timeout, the job fails. So a job asking for 8 nodes on a pool with MAX_NODES = 4 never gets them and fails with an error. It doesn't quietly train on 4.
@remote(
"MY_COMPUTE_POOL",
stage_name="payload_stage",
session=session,
target_instances=5, # Prefer 5 nodes
min_instances=3 # But start with at least 3 nodes
)min_instances lets a job start with fewer nodes. The payload starts as soon as that minimum is available, even if it's below target_instances. That cuts waiting on a busy pool and lets a workflow cope with whatever capacity it gets. The payload then has to adapt. A common pattern counts ray.nodes() and passes the count to XGBScalingConfig(num_workers=...).
To debug, read a specific node's log with job.get_logs(instance_id=1). Calling it without an ID returns the head node's log. Instance 0 is not necessarily the head node, so don't assume instance_id=0 gives you the coordinator.
Checkpoint 4 of 6· Check yourself
An ML Job is submitted with target_instances=8 to a compute pool created with MAX_NODES = 4. No min_instances is set. What happens?
Before it runs the payload, an ML Job waits until target_instances nodes are ready. A pool capped at 4 nodes can never provide 8, so the job fails at the timeout and does not quietly train on 4. Only min_instances lets a job start with fewer nodes than the target.
“The job fails with an error if the expected nodes aren’t available within the timeout period.”Source: docs.snowflake.com
Checkpoint 5 of 6· Fill the gap
On a busy pool, a distributed training job should not sit waiting for all five nodes. It should start once three are ready. Which parameter completes the decorator?
@remote(
"MY_COMPUTE_POOL",
stage_name="payload_stage",
session=session,
target_instances=5, # Prefer 5 nodes
? =3 # But start with at least 3 nodes
)With min_instances set, the payload starts as soon as that many nodes are available, even if that is fewer than target_instances. MIN_NODES is a compute pool setting, not a job parameter. num_workers belongs to XGBScalingConfig inside the payload.
Source: docs.snowflake.comSources2
4.Custom runtime images for ML Jobs
A Custom Runtime Environment (CRE) is a container image registered with Snowflake so that ML Jobs and Notebooks run in it instead of the stock Container Runtime image. The path from a built image to a running job has three gates: local validation, server-side registration, and a digest check at submission.
Build for linux/amd64 from an official Snowflake ML runtime base image, which comes per hardware type (CPU or GPU) and Python version. Then check the image locally with snow custom-image validate. This check is advisory, but it catches problems before Snowflake rejects the image at registration. To pass:
- the entrypoint must be exactly
/usr/local/bin/entrypoint.sh - the environment must set
DASHBOARD_PORT=12003 - the mandatory packages must be installed, including
snowflake-ml-python,rayandipykernel - the dependency tree must resolve with no conflicts
Add --scan-vulnerabilities to also scan for CVEs. That scan isn't required to pass.
CREATE CUSTOM RUNTIME ENVIRONMENT my_cre
IMAGE_PATH = '/<database>/<schema>/<image_repo>/my-custom-image:v1'
BASE_IMAGE_TYPE = CPU;After you tag the image and push it to a Snowflake image repository, register it with CREATE CUSTOM RUNTIME ENVIRONMENT. Snowflake runs a server-side check that the image actually executes, and creates the CRE only if that check passes. The CRE is then in the RESOLVED state. To use it, a job passes runtime_environment="cre@my_cre". That needs snowflake-ml-python 1.37.0 or later, a role with USAGE on the CRE plus access to the image repository, and a CRE that is not DEPRECATED. Registered CREs can also be chosen for notebooks in Workspaces, where notebook runtimes are set up.
A CRE is tied to one exact image, not to a tag. When a job is submitted, Snowflake checks the image's SHA digest. If someone pushed a changed image to the same path after registration, the submission is rejected. To ship a fixed image, create the CRE again.
Checkpoint 6 of 6· Put it in order
Put these steps in order to run an ML Job in your own image, which has libraries the stock Container Runtime lacks.
- 1.Submit the job with runtime_environment="cre@my_cre"
- 2.Check the image locally with snow custom-image validate
- 3.Tag the image with the Snowflake image repository path and push it
- 4.Write a Dockerfile FROM an official Snowflake ML runtime base image and build it for linux/amd64
- 5.Register the image with CREATE CUSTOM RUNTIME ENVIRONMENT, which runs a server-side check
The image must be built from a Snowflake base image before it can be checked. The local check comes before the push and catches problems early. Registration needs the image to be in a Snowflake repository already, and a job can only reference a CRE that is registered and in the RESOLVED state.
“Before pushing the image to Snowflake, validate it locally to ensure it meets runtime requirements.”Source: docs.snowflake.com
Sources3
Practise it for real
Turn a pinned-library Docker image into a Custom Runtime Environment and run an ML Job in it
1.Run
snow spcs image-registry login, thendocker pullthe CPU base imagecontainer_runtime/cpu_x86_64:2.7.2from your registry URL.Why: Every custom image must derive from an official Snowflake ML runtime base image, and Docker has to be logged in to the Snowflake registry to pull it.
You should see: The base image appears in
docker images.2.Write a Dockerfile
FROMthat base image that installsxgboost==2.0.3andscikit-learn==1.3.2, then rundocker build --platform linux/amd64 -t my-custom-image:v1 .Why: Snowflake runs linux/amd64 images, and pinning versions makes the environment reproducible.
You should see: A local image tagged
my-custom-image:v1.3.Run
snow custom-image validate my-custom-image:v1 --scan-vulnerabilities.Why: Validating locally catches a wrong entrypoint, a missing DASHBOARD_PORT, missing mandatory packages or dependency conflicts before registration fails on the server.
You should see: A validation report with no blocking errors. CVE findings are advisory.
4.Tag the image with
<registry_url>/<database>/<schema>/<image_repo>/my-custom-image:v1,docker pushit, then runCREATE CUSTOM RUNTIME ENVIRONMENT my_crewithIMAGE_PATHandBASE_IMAGE_TYPE = CPU.Why: Registration runs the server-side check and ties the CRE to this exact image digest.
You should see: The CRE is created in the RESOLVED state.
5.Decorate a function with
@jobs.remote("MY_COMPUTE_POOL", stage_name="payload_stage", runtime_environment="cre@my_cre"), call it, then calljob.result().Why: This shows the job ran in your image and not in the default Container Runtime.
You should see: The job reaches DONE and
result()returns the function's value.
Stuck? Get a nudge
If submission is rejected after you rebuilt and re-pushed the same tag, the digest check is working as designed. Create the CRE again.
Sources
(3)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Snowflake ML Jobs are available in snowflake-ml-python version 1.26.0 and later.”
↩︎ Submitting work as an ML Job“Snowflake automatically uses the latest available version of the Snowflake Container Runtime on your compute pool.”
↩︎ Submitting work as an ML Job“You can only load single Python files from a stage.”
↩︎ Submitting work as an ML Job“The API blocks the calling thread until the job reaches a terminal state”
↩︎ Monitoring jobs and getting results“For file-based jobs, use the special __return__ variable to specify the return value.”
↩︎ Monitoring jobs and getting results“For file-based jobs, use the special __return__ variable to specify the return value.”
↩︎ Exam trap 3“To specify a container runtime for your ML Job, use the runtime_environment keyword with the full Container Runtime version string.”
↩︎ Checkpoint“if execution failed, raises an exception.”
↩︎ Prediction - 2.
“For multi-node jobs, your code needs to be designed for distributed processing using Distributed Modeling Classes or Ray.”
↩︎ Multi-node ML Jobs“The job fails with an error if the expected nodes aren’t available within the timeout period.”
↩︎ Multi-node ML Jobs“the job payload is executed as soon as the minimum number of nodes becomes available”
↩︎ Multi-node ML Jobs“The job fails with an error if the expected nodes aren’t available within the timeout period.”
↩︎ Exam trap 1 - 3.
“All custom images must be derived from an official Snowflake ML runtime base image.”
↩︎ Custom runtime images for ML Jobs“During this step, Snowflake performs a server-side validation check to confirm the image executes correctly.”
↩︎ Custom runtime images for ML Jobs“The active role must hold the USAGE privilege on the target Custom Runtime Environment, along with access to the underlying image repository.”
↩︎ Custom runtime images for ML Jobs“If the underlying image in the repository was modified after the CRE was registered, the job submission is rejected to ensure reproducibility.”
↩︎ Exam trap 2“Before pushing the image to Snowflake, validate it locally to ensure it meets runtime requirements.”
↩︎ Checkpoint