What you will be able to do
- Choose between a larger standard warehouse and a Snowpark-optimized warehouse for a memory-bound ML job
- Apply auto-suspend, multi-cluster scaling, timeouts and resource monitors to limit warehouse spend
- Explain which compute pool states are billed and what sets a pool's credit rate
- Track compute pool credits in the metering views and cut them by right-sizing, capping nodes and auto-suspending
1.Warehouse size and type for ML jobs
Warehouse size buys compute, and memory is what most often slows ML jobs down. A bigger warehouse gives a query more compute resources. A query that runs out of memory spills bytes to storage and becomes much slower, so you pay for longer runs. When the query profile of a training or feature job shows heavy spilling, fix the memory problem before anything else.
Snowpark-optimized warehouses are designed for single-node ML training in stored procedures and for code that needs a specific CPU architecture. RESOURCE_CONSTRAINT sets memory per node, and each memory tier has a minimum warehouse size. Pick them by workload, not by default. SQL-only work that doesn't use Snowpark might gain nothing from the optimized type. These warehouses also take longer to create and resume than standard ones. A pool of similar workloads is easier to tune, because a change made for one query is not wasted on unrelated queries sharing the warehouse.
| RESOURCE_CONSTRAINT values | Memory (up to) | Minimum warehouse size |
|---|---|---|
| MEMORY_1X, MEMORY_1X_x86 | 16GB | XSMALL |
| MEMORY_16X, MEMORY_16X_x86 (default) | 256GB | M |
| MEMORY_64X, MEMORY_64X_x86 | 1TB | L |
CREATE WAREHOUSE so_warehouse WITH
WAREHOUSE_SIZE = 'LARGE'
WAREHOUSE_TYPE = 'SNOWPARK-OPTIMIZED'
RESOURCE_CONSTRAINT = 'MEMORY_16X_X86';Checkpoint 1 of 5· Fill the gap
A Snowpark training procedure needs up to 256 GB of memory per node on x86 CPUs. Which RESOURCE_CONSTRAINT value completes this LARGE Snowpark-optimized warehouse?
CREATE WAREHOUSE so_warehouse WITH
WAREHOUSE_SIZE = 'LARGE'
WAREHOUSE_TYPE = 'SNOWPARK-OPTIMIZED'
RESOURCE_CONSTRAINT = ' ? ';MEMORY_16X gives up to 256 GB, and the _X86 suffix pins the x86 architecture. MEMORY_64X is the 1 TB tier and MEMORY_1X is the 16 GB tier. Plain MEMORY_16X leaves the CPU architecture at the default instead of x86. LARGE meets the M minimum size for the 16X tier.
Source: docs.snowflake.com2.Suspend, scale out and cap: warehouse cost controls
A warehouse costs nothing while it is suspended, so auto-suspend is the biggest single saving for bursty ML work like nightly feature builds. Auto-suspend is on by default. Pair it with auto-resume so the warehouse restarts when the next job arrives. Then check from time to time that nobody has turned auto-suspend off.
SHOW WAREHOUSES
->> SELECT "name" AS WAREHOUSE_NAME,
"size" AS WAREHOUSE_SIZE
FROM $1
WHERE IFNULL("auto_suspend", 0) = 0;Upsizing is the usual way a warehouse's cost creeps up. A data scientist raises a warehouse to LARGE for one heavy experiment and never sets it back, and every later job pays the larger rate. The MODIFY privilege controls who can resize a warehouse or turn off its auto-suspend. Granting it, and CREATE WAREHOUSE, to only a small central role prevents this. When load rises and falls, for example many analysts querying features at once, a multi-cluster warehouse scales out and back by itself, with no manual resize to forget.
Three more controls limit runaway spend:
STATEMENT_TIMEOUT_IN_SECONDScancels hung queries, which waste credits by running longer than expected. It can be set per account, user, session or warehouse.STATEMENT_QUEUED_TIMEOUT_IN_SECONDScancels statements that wait in the queue too long. Queued statements cost nothing, but one that runs after its result is no longer useful wastes credits.- Resource monitors set a credit limit for a period. When the limit is reached they can notify an administrator, or suspend the warehouse either after running statements finish or immediately.
Checkpoint 2 of 5· Check yourself
At peak hours, dozens of analysts query feature tables on a shared ML warehouse at once, and queries queue. A team lead suggests giving the data scientists MODIFY on the warehouse so they can upsize it when it gets busy. What is the more cost-safe approach?
Users who upsize a warehouse often forget to size it back down, so every later job pays the larger rate. Keeping MODIFY with a central role prevents that, and a multi-cluster warehouse absorbs changing concurrency on its own. A resource monitor caps spend but does nothing about the load, and a queue timeout just cancels the analysts' work.
“consider using a multi-cluster warehouse. A multi-cluster warehouse scales automatically as workloads fluctuate.”Source: docs.snowflake.com
Sources3
3.What a compute pool bills for
A compute pool's credit rate comes from its nodes: how many there are and which instance family they use (Compute pools). Choosing an instance family is like choosing a warehouse size. Each family sets the vCPU, memory, storage and egress bandwidth of a node. A GPU family therefore costs much more per hour than a small CPU family, whether or not the GPU is busy.
| State | Billed |
|---|---|
| IDLE | Yes |
| ACTIVE | Yes |
| STOPPING | Yes |
| RESIZING | Yes |
| STARTING | No |
| SUSPENDED | No |
An IDLE pool is billed at the full rate for its nodes. A GPU pool left idle over a weekend is pure waste, and auto-suspend is the control that stops it. Containers add a few smaller costs on top of compute. The image repository and any stage mounted as a volume are billed as stage storage. Container logs written to an event table are billed as table storage. Block storage volumes are billed together with their snapshots. Outbound data transfer is charged at the standard Snowflake rate.
Checkpoint 3 of 5· Check yourself
Over a long weekend, a GPU compute pool that was never suspended runs no services or jobs. What does it cost for those days, and what would have prevented the cost?
IDLE is a billed state. A pool is charged for its nodes until it reaches SUSPENDED, and the instance family sets the rate whether or not the GPU is busy. Auto-suspend is the control that ends this waste.
“incur charges for a compute pool in the IDLE, ACTIVE, STOPPING, or RESIZING state, but not when it is in a STARTING or SUSPENDED state”Source: docs.snowflake.com
4.Tracking compute pool credits and cutting them
SNOWPARK_CONTAINER_SERVICES_HISTORY is the dedicated compute pool view. It holds hourly CREDITS_USED per COMPUTE_POOL_NAME for the last year, so a per-pool or per-team monthly total is a simple GROUP BY. The view can lag by up to three hours, so don't build a same-hour alert on it. For a combined view of all services, filter METERING_HISTORY or METERING_DAILY_HISTORY on service_type = SNOWPARK_CONTAINER_SERVICES. The ORGANIZATION_USAGE version of METERING_DAILY_HISTORY accepts the same filter.
Once you know which pool costs the most, there are three levers. Use them in this order:
- Right-size the instance family. Take a real-time scoring service on GPU nodes that averages 5 percent GPU use while serving a small gradient-boosted tree. It belongs on a CPU family. Run
SHOW COMPUTE POOL INSTANCE FAMILIESto see what your region offers. - Lower
MAX_NODES. The maximum stops autoscaling from adding many nodes during a load spike or because of a bug in your code. KeepMIN_NODESabove 1 only for services that must absorb sudden bursts without waiting for autoscaling. - Set
AUTO_SUSPEND_SECS. The pool suspends after that many seconds with no services and no jobs on any node. WithAUTO_RESUMEon, the next job submitted to the pool resumes it.
CREATE COMPUTE POOL tutorial_compute_pool MIN_NODES = 1 MAX_NODES = 1 INSTANCE_FAMILY = CPU_X64_XS;Checkpoint 4 of 5· Check yourself
Each ML team runs its Snowpark Container Services work on its own compute pool. Finance asks for last month's container compute credits per team. Where should you get the numbers?
Compute pool credits are recorded per pool, by the hour, in SNOWPARK_CONTAINER_SERVICES_HISTORY. With one pool per team, a GROUP BY on COMPUTE_POOL_NAME gives the per-team total. The warehouse views and per-query views don't hold compute pool credits, and this view can lag by up to three hours, so it can't drive a same-hour alert.
“The SNOWPARK_CONTAINER_SERVICES_HISTORY view in the ACCOUNT_USAGE schema can be used to return the hourly compute pool credit usage for an account”Source: docs.snowflake.com
Checkpoint 5 of 5· Fill the gap
Load spikes or a bug in your code could make autoscaling add far more nodes than planned. Which parameter in this statement prevents that?
CREATE COMPUTE POOL tutorial_compute_pool MIN_NODES = 1 ? = 1 INSTANCE_FAMILY = CPU_X64_XS;MAX_NODES sets the most nodes Snowflake autoscaling can add to the pool, so it caps spend during load spikes or runaway code. AUTO_SUSPEND_SECS controls billing while the pool is IDLE. MAX_CLUSTER_COUNT is a multi-cluster warehouse setting, not a compute pool one. MIN_NODES sets the floor, not the ceiling.
Source: docs.snowflake.comSources
(8)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“a query runs substantially slower when a warehouse runs out of memory, which results in bytes “spilling” onto storage.”
↩︎ Warehouse size and type for ML jobs“Optimizing a warehouse for query performance is more straightforward when the warehouse runs similar workloads.”
↩︎ Warehouse size and type for ML jobs - 2.
“Example workloads include Machine Learning (ML) training use cases using a stored procedure on a single virtual warehouse node.”
↩︎ Warehouse size and type for ML jobs“Workloads that don’t use Snowpark might not benefit from running on Snowpark-optimized warehouses.”
↩︎ Warehouse size and type for ML jobs“The default configuration for a Snowpark-optimized warehouse provides 16x memory per node compared to a standard warehouse.”
↩︎ Prediction - 3.
“A suspended warehouse does not consume credits, so the warehouse only incurs cost when it is processing a workload.”
↩︎ Suspend, scale out and cap: warehouse cost controls“A multi-cluster warehouse scales automatically as workloads fluctuate.”
↩︎ Suspend, scale out and cap: warehouse cost controls“Hung queries consume excessive credits because they run longer than expected.”
↩︎ Suspend, scale out and cap: warehouse cost controls“the warehouse is suspended when either of the credit limits is reached.”
↩︎ Suspend, scale out and cap: warehouse cost controls“Commonly, users increase the size of a warehouse for a particular workload and then forget to change it back to its original size”
↩︎ Exam trap 2“consider using a multi-cluster warehouse. A multi-cluster warehouse scales automatically as workloads fluctuate.”
↩︎ Checkpoint - 4.https://docs.snowflake.com/en/developer-guide/snowpark-container-services/accounts-orgs-usage-viewsOfficial docs
“The number and type (instance family) of the nodes in the compute pool (see CREATE COMPUTE POOL) determine the credits it consumes”
↩︎ What a compute pool bills for“but not when it is in a STARTING or SUSPENDED state”
↩︎ What a compute pool bills for“In the METERING_DAILY_HISTORY view, query for rows in which the service_type column contains the value SNOWPARK_CONTAINER_SERVICES.”
↩︎ Tracking compute pool credits and cutting them“You incur charges for a compute pool in the IDLE, ACTIVE, STOPPING, or RESIZING state”
↩︎ Exam trap 1“incur charges for a compute pool in the IDLE, ACTIVE, STOPPING, or RESIZING state, but not when it is in a STARTING or SUSPENDED state”
↩︎ Checkpoint - 5.https://docs.snowflake.com/en/developer-guide/snowpark-container-services/instance-familiesOfficial docs
“An instance family specifies the vCPU, memory, storage, and egress bandwidth available to nodes in a compute pool.”
↩︎ What a compute pool bills for - 6.https://docs.snowflake.com/en/sql-reference/account-usage/snowpark_container_services_historyOfficial docs
“Latency for the view may be up to 180 minutes (3 hours).”
↩︎ Tracking compute pool credits and cutting them“The SNOWPARK_CONTAINER_SERVICES_HISTORY view in the ACCOUNT_USAGE schema can be used to return the hourly compute pool credit usage for an account”
↩︎ Checkpoint - 7.https://docs.snowflake.com/en/developer-guide/snowpark-container-services/working-with-compute-poolOfficial docs
“Setting a maximum node limit prevents an unexpectedly large number of nodes from being added to your compute pool by Snowflake autoscaling.”
↩︎ Tracking compute pool credits and cutting them - 8.
“Number of seconds of inactivity after which you want Snowflake to automatically suspend the compute pool.”
↩︎ Tracking compute pool credits and cutting them