Storing and Loading ML Training and Inference Data: Tables, Stages and Data Shares

1.Tables: load with DataConnector, write with DataSink

When training or inference data sits in Snowflake tables, Snowflake ML provides one API to read it and one to write it. Snowflake Notebooks and Snowflake ML Jobs both run on Container Runtime, which uses Ray to process data across multiple compute nodes. In that environment, DataConnector loads structured data from tables and Snowflake Datasets, and DataSink writes structured data back to tables.

DataConnector speeds up loading by parallelizing reads across the compute nodes. The docs say it outperforms to_pandas on large datasets. The loaded data can become a pandas DataFrame, a PyTorch dataset or a TensorFlow dataset. For the best performance, you can instead pass the connector directly to Snowflake's distributed training APIs.

A connector can be built from either source, and the docs pair each with a stage of the lifecycle. A Snowpark DataFrame gives direct access to the table and is best used during development. A Snowflake Dataset is a versioned, schema-level object and is best used for production workflows.

from snowflake.ml.data.data_connector import DataConnector
from snowflake.snowpark.context import get_active_session

session = get_active_session()

# Create DataConnector from a Snowflake table
data_connector = DataConnector. ? (session.table("example-table-name"))

Sources12

2.Internal and external stages for file-based data

Data that arrives as files, such as CSV, Parquet or images, lives in a stage. Snowflake has two kinds. An internal stage stores the files inside Snowflake. An external stage points to files outside Snowflake, in Amazon S3 buckets, Google Cloud Storage buckets or Microsoft Azure containers, and the location can be private/protected or public. External stages have one catch: data in archival storage classes that need restoring before retrieval, such as S3 Glacier Deep Archive, can't be read through them.

You can read staged files in two ways. In Snowpark, session.read plus a format method turns a staged file into a DataFrame, which you can then feed into the preprocessors and transformations above. In Container Runtime, the DataSource APIs load CSV, Parquet, images and other formats from stages. Two optimizations help when reading staged files. Stage sharding consolidates many small files into larger shards to cut per-file I/O. A disk cache, enabled by default, serves repeated reads from node-local disk rather than fetching again from Snowflake.

df_json = session.read.json("@my_stage2/data1.json")

Internal stages also have a cost angle. Files in an internal stage incur standard data storage costs but none of the extra Time Travel and Fail-safe costs. Snowflake recommends removing staged files once they are loaded and no longer needed, either during COPY INTO or afterwards with REMOVE. Regular purging can also improve data loading performance.

Sources314

3.Data shares for training data owned by another account

When training data belongs to another Snowflake account, Secure Data Sharing gives you access without moving the data. A provider creates a share containing selected objects from one of their databases, and the consumer imports it. Shareable objects include tables, dynamic tables, external tables, Iceberg tables, regular and secure views, secure materialized views, and UDFs.

No data is copied between accounts, because sharing runs through Snowflake's services layer and metadata store. Two consequences follow for ML data. The shared data takes up no storage in the consumer account, so the consumer pays only for the compute that queries it. And access is near-instant once the share is set up. The governance trade-off is control: every shared object is read-only for the consumer. You can query a shared table to build features, but you can't modify it or add data to it.

Sources5

4.Matching storage to performance, cost and governance

For a given training or inference dataset, choose the storage method by the requirement that matters most. The table below collects the deciding facts for each option. It covers only what the sources here state. In particular, they say little about governance controls on ordinary tables, so that cell describes how the data is accessed rather than which policies apply.

Sources3567

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Consuming a data share copies the provider's tables into your account, so you pay storage for them.Why is that wrong?

   Sharing copies nothing. Shared data takes up no storage in the consumer account, and the consumer pays only for the warehouses that query it.

   Covered in Data shares for training data owned by another account

2. 2.Files left in an internal stage carry Time Travel and Fail-safe costs, just like table data.Why is that wrong?

   Internally staged files incur only standard storage costs. Snowflake still recommends removing them after loading to control cost and improve loading performance.

   Covered in Internal and external stages for file-based data

3. 3.A DataConnector over a live Snowpark DataFrame is the recommended input for production training.Why is that wrong?

   The docs pair Snowpark DataFrames with development. For production workflows they recommend Snowflake Datasets, which are versioned.

   Covered in Tables: load with DataConnector, write with DataSink