CertSafari
    Snowflake SnowPro Advanced: MLOps Engineer (MLA-B01)· Lessons

    Domain 1 · Lesson 1/17

    Distributed feature engineering with Snowpark and Snowflake ML

    Construct distributed feature engineering pipelines.

    16 min read
    4% of exam
    7 sources
    Published 20 Oct 2026
    Docs as of 4 Oct 2026

    What you will be able to do

    • Build lazy Snowpark DataFrames from tables, stages and SQL, and know which call actually runs the query
    • Fit and apply Snowflake ML preprocessors and Pipelines on large tables without pulling rows to the client
    • Decide whether a transformation belongs in plain SQL, Snowpark DataFrames, a UDF or Snowflake ML preprocessors
    • Choose tables, internal or external stages for training and inference data, and write results back with DataSink

    Key concept

    Pushdown — Snowpark DataFrame operations and Snowflake ML preprocessors become SQL that the warehouse runs where the data already lives. Your Python process only describes the work. The rows stay in Snowflake, so data size is the warehouse's problem and not your client's.

    1.Snowpark DataFrames build queries, not result sets

    Feature engineering on a billion-row table only works if the table never has to fit in your notebook's memory. Snowpark handles this with the DataFrame. A Snowpark DataFrame is a description of a relational query. It is not a copy of the data. Selecting columns, filtering rows, grouping and sorting all add to that description, and nothing reaches the warehouse until you call an action such as collect().

    Constructing a DataFrame runs nothing; collect() sends the SQL and returns Rowspython
    >>> # Create a DataFrame with the "id" and "name" columns from the "sample_product_data" table.
    >>> # This does not execute the query.
    >>> df = session.table("sample_product_data").select(col("id"), col("name"))
    
    >>> # Send the query to the server for execution and
    >>> # return a list of Rows containing the results.
    >>> results = df.collect()

    Laziness is what makes long feature pipelines cheap. Every join, filter and derived column you chain up is batched into one operation, which runs as late as possible. Only the final result crosses the network, and only if you ask for it. If an action like collect() or show() sits in the middle of the pipeline, it forces a round trip there. If a final collect() runs on a wide training table, every row comes back to the client, and that cancels the benefit.

    The Session object gives you a different constructor for each kind of source:

    • session.table(...) for a table, view or stream
    • session.read.json(...) or session.read.schema(...).csv(...) for files in a stage, optionally with an explicit StructType schema
    • session.sql(...) for the result of a SQL statement
    • session.create_dataframe(...) and session.range(...) for small local values, which are useful in tests

    Checkpoint 1 of 7· Fill the gap

    The first line only builds a lazy DataFrame. Which method completes the last line so the SQL is sent to Snowflake and the results come back as a list of Rows?

    >>> # Create a DataFrame with the "id" and "name" columns from the "sample_product_data" table.
    >>> # This does not execute the query.
    >>> df = session.table("sample_product_data").select(col("id"), col("name"))
    
    >>> # Send the query to the server for execution and
    >>> # return a list of Rows containing the results.
    >>> results = df. ? ()

    Sources12

    2.Snowflake ML preprocessors and Pipelines run in the warehouse

    Scaling and encoding have a fit step, and that is the hard part to distribute. A scaler such as MinMaxScaler must first learn each column's minimum and maximum. An encoder such as OneHotEncoder must first learn the set of categories. In scikit-learn, fit scans a pandas DataFrame in local memory. The classes in snowflake.ml.modeling.preprocessing take a Snowpark DataFrame instead and push the work down to the warehouse your session is using. Statistics like a column's minimum and maximum are computed inside Snowflake, so the rows never leave it.

    Two terms in the comparison below need defining. Container Runtime is a preconfigured environment for ML workloads, used by Snowflake Notebooks and ML Jobs. It runs on compute pools (Snowpark Container Services) instead of a warehouse, and it uses Ray, an open-source framework for distributed computing, to spread Python across nodes. Ray map_batches is the Ray call that applies your own Python function to batches of data in parallel. It is lazy, so nothing runs until the dataset is materialized. A warehouse, by contrast, runs SQL that Snowflake generates for you.

    Three ways to engineer features in Snowflake ML, by where the work runs
    AspectOSS (scikit-learn)Snowflake ML preprocessorsRay map_batches
    ScaleSmall and medium datasetsLarge/distributed dataLarge/distributed data
    ExecutionIn memoryPushdown to the default warehouse running your SQL queriesAcross nodes in a compute pool
    ComputeSnowpark Container Services (Compute Pool)WarehouseSnowpark Container Services (Compute Pool)
    Best forQuick prototyping and experimentationProduction workflows with large datasetsCustom logic, unstructured data, custom resource controls

    The table rules out the usual wrong answers about where a preprocessor's fit runs. In-memory execution belongs to the open-source column. Compute pools belong to scikit-learn on Container Runtime and to Ray. A Snowflake ML preprocessor uses neither: it does not provision a container job, and it does not stream rows through the Python driver. It sends SQL to the warehouse. That also makes warehouse size the setting that matters for these preprocessors.

    A Pipeline of Snowflake ML preprocessors fitted and applied on a table-backed DataFramepython
    # Load your data from a Snowflake table as a DataFrame
    df = session.table('CUSTOMER_DATA')
    
    # Define Snowflake ML preprocessors
    scaler = StandardScaler(input_cols=['AGE', 'INCOME'], output_cols=['AGE_SCALED', 'INCOME_SCALED'])
    encoder = OneHotEncoder(input_cols=['CITY'], output_cols=['CITY_ENCODED'])
    
    pipeline = Pipeline(steps=[
        ('scaling', scaler),
        ('encoding', encoder)
    ])
    
    # Fit and transform data in Snowflake (distributed)
    result = pipeline.fit_transform(df)
    result.show()

    Compared with scikit-learn, the visible API change is that columns are named on the transformer itself. You pass input_cols and output_cols, and you don't need a ColumnTransformer to route columns. Pipeline comes from snowflake.ml.modeling.pipeline and chains steps exactly as its scikit-learn namesake does. fit_transform returns another Snowpark DataFrame, which means the result is still lazy until show() or another action runs. A fitted pipeline holds what it learned from the data it was fitted on, such as ranges and category lists. Calling fit again on different data replaces those learned values.

    The catalogue is smaller than scikit-learn's. Snowflake ML ships a subset of the scikit-learn preprocessors that covers the common cases, including StandardScaler, MinMaxScaler and OneHotEncoder. To see the parameters of any of them, import it and call help(OneHotEncoder) in a Python session.

    Checkpoint 2 of 7· Check yourself

    A production job fits a Snowflake ML StandardScaler and OneHotEncoder Pipeline on a billion-row table-backed Snowpark DataFrame, and the fit is too slow. Which change speeds up the fit itself?

    Checkpoint 3 of 7· Exam question

    A team stores 2 billion customer rows in a Snowflake table. A data scientist calls fit on a snowflake.ml.modeling.preprocessing.MinMaxScaler using a Snowpark DataFrame built from that table. Where are the per-column minimum and maximum values computed?

    Checkpoint 4 of 7· Exam question

    Which Python module contains the scikit-learn-style distributed preprocessors such as MinMaxScaler, OneHotEncoder, and OrdinalEncoder that execute inside Snowflake?

    Sources3425

    3.Choosing SQL, Snowpark or ML preprocessors for a transformation

    Plain SQL and Snowpark DataFrames both run in the warehouse, so speed rarely decides between them. The Snowpark API provides Python constructs for building SQL statements: a DataFrame select produces the same kind of query you would otherwise write as a string. For set-based work like filters, joins and aggregations, both execute as warehouse SQL. Choose based on how complex the logic is, whether it will be reused, and who will maintain it.

    Usually not. session.sql(...) wraps the existing query in a lazy DataFrame. From there the engineer can join it, pass it to a preprocessor or write it out, while the analysts keep owning SQL they already understand. Snowpark DataFrame methods win when the transformation is built in code: logic shared across pipelines as Python functions, column lists generated from configuration, or steps that branch on parameters. Native methods such as session.table and session.read also give better syntax highlighting, error highlighting and code completion than SQL inside a string.

    Matching a transformation to the tool that should express it
    ToolWins whenWhere it runs
    Plain SQL (session.sql or a SQL worksheet)Set-based logic that SQL-fluent analysts own and maintainWarehouse
    Snowpark DataFrame methodsLogic composed and reused in Python code, with completion and type checkingWarehouse, as generated SQL
    Snowpark UDFCustom row-level or looping Python logic that SQL cannot expressWarehouse, where the data is
    Snowflake ML preprocessorsStandard scaling or encoding that needs a fit step over a large tableWarehouse
    Ray map_batchesCustom large-scale logic, especially on unstructured dataCompute pool nodes

    UDFs fill the gap between SQL and full Python. If a feature needs procedural logic, write the function in Python and register it as a UDF. Snowpark pushes the code to the server and runs it in parallel next to the data, so there is no need to pull rows out to loop over them. Don't hand-write min-max scaling in SQL or a UDF, though. Fitting and applying a scaler or encoder across a big table is exactly the job Snowflake ML preprocessors are built for.

    Checkpoint 5 of 7· Match them up

    Match each transformation need to the tool that should express it in a Snowflake feature pipeline.

    Tap a term, then the definition that fits it.

    Sources213

    4.Storing training and inference data: tables, stages and write-back

    Where features and scoring inputs live determines how they load, what they cost and who can read them. Structured data that you query, join and transform belongs in Snowflake tables. That is where Snowpark DataFrames and preprocessors do their work, and where results are written back. Stages hold files. An internal stage keeps them inside Snowflake. An external stage points at Amazon S3, Google Cloud Storage or Microsoft Azure. Raw files, Parquet exports and unstructured data like images typically land in a stage.

    Loading and writing APIs by where the data lives
    Data sourceData typeLoad withWrite with
    Snowflake TablesStructuredDataConnectorDataSink
    Snowflake DatasetsStructuredDataConnectorDataSink
    CSV or Parquet Files (Stage)StructuredDataSource APIDataSink
    Other Staged FilesUnstructuredDataSource APIN/A

    The two loading APIs split by where the data sits. DataConnector loads structured data from Snowflake tables and Snowflake Datasets. The DataSource API loads files from Snowflake stages in formats such as CSV, Parquet and images. Reach for DataConnector when the data is already a table or Dataset, and for the DataSource API when it is staged files, which is the only loading path for unstructured files like images.

    DataConnector is the shared loading interface for structured data, whether it comes from a table, a Dataset or shared data. That means the storage choice affects load performance and governance but not the loading code. It speeds up loading by spreading reads across multiple compute nodes. A DataConnector built from a Snowpark DataFrame on a table reads that table directly, which suits development. For production training, the guidance is a versioned Snowflake Dataset. Datasets carry their own storage cost and their own privileges: reading one requires USAGE on that Dataset.

    With staged files, performance depends on layout. Partition stage paths by source and date, for example United_States/California/Los_Angeles/2016/06/01/11/. Point each load at the most specific path that contains the data, so it lists and reads fewer files and parallel loads can each take a separate slice. If a stage holds thousands of small files, stage sharding merges them into larger shards to reduce per-file I/O. A node-local disk cache, on by default, serves repeated reads without fetching from Snowflake again.

    An external stage scoped to one partition path of an S3 bucketsql
    CREATE STAGE my_stage URL='s3://mybucket/United_States/California/Los_Angeles/' CREDENTIALS=(AWS_KEY_ID='1a2b3c' AWS_SECRET_KEY='4x5y6z');

    Writing results back follows the same split. DataSink writes structured output, such as engineered features or batch predictions, into Snowflake tables, where they can be governed and queried like any other table. Unstructured staged files have a loading API but no DataSink path back. Once processed, their output has to be stored as structured rows.

    Checkpoint 6 of 7· Check yourself

    An inference job uses the DataSource API to load product photos from an internal stage and classifies each image. Where and how should the job persist its output?

    Checkpoint 7 of 7· Exam question

    A fraud model was trained with a snowflake.ml.modeling.preprocessing.OneHotEncoder fitted on the merchant_category column. During batch inference the encoder raises an error because new merchant categories appeared that were absent from the training data. Which change allows scoring to continue without refitting?

    Sources657

    Sources

    (7)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “A DataFrame represents a relational dataset that is evaluated lazily: it only executes when a specific action is triggered.”
      ↩︎ Snowpark DataFrames build queries, not result sets
      “To create a DataFrame from data in a table, view, or stream, call the table method”
      ↩︎ Snowpark DataFrames build queries, not result sets
      “using the table method and read property offer better syntax highlighting, error highlighting, and intelligent code completion in development tools”
      ↩︎ Choosing SQL, Snowpark or ML preprocessors for a transformation
      “A DataFrame represents a relational dataset that is evaluated lazily: it only executes when a specific action is triggered.”
      ↩︎ Exam trap 1
    2. 2.
      “delay running data transformation until as late in the pipeline as possible while batching up many operations into a single operation”
      ↩︎ Snowpark DataFrames build queries, not result sets
      “build applications that process data in Snowflake without moving data to the system where your application code runs”
      ↩︎ Snowflake ML preprocessors and Pipelines run in the warehouse
      “The Snowpark API provides programming language constructs for building SQL statements.”
      ↩︎ Choosing SQL, Snowpark or ML preprocessors for a transformation
      “useful for looping or batch functionality where creating as a UDF will allow Snowflake to parallelize”
      ↩︎ Choosing SQL, Snowpark or ML preprocessors for a transformation
      “Snowpark pushes down all data transformation and heavy lifting to the Snowflake data cloud”
      ↩︎ Key concept
      “The data isn’t retrieved when you construct the DataFrame object.”
      ↩︎ Prediction
      “useful for looping or batch functionality where creating as a UDF will allow Snowflake to parallelize and apply the codeful logic at scale”
      ↩︎ Checkpoint
    3. 3.
      “Snowflake ML preprocessors handle distributed transformations directly within Snowflake.”
      ↩︎ Snowflake ML preprocessors and Pipelines run in the warehouse
      “Pushdown to the default warehouse that you’re using to run SQL queries”
      ↩︎ Snowflake ML preprocessors and Pipelines run in the warehouse
      “Use Snowflake ML preprocessors for large datasets and production workloads.”
      ↩︎ Choosing SQL, Snowpark or ML preprocessors for a transformation
      “Pushdown to the default warehouse that you’re using to run SQL queries”
      ↩︎ Exam trap 2
      “The Snowflake ML preprocessors are a subset of the preprocessors available in scikit-learn, but they cover the most common use cases.”
      ↩︎ Exam trap 3
    4. 5.
      “The Container Runtime uses Ray, an open-source framework for distributed computing, to efficiently process data across multiple compute nodes.”
      ↩︎ Snowflake ML preprocessors and Pipelines run in the warehouse
      “Cut per-file I/O when reading many small files by consolidating them into larger shards.”
      ↩︎ Storing training and inference data: tables, stages and write-back
      “DataSink: Write data back to Snowflake tables.”
      ↩︎ Storing training and inference data: tables, stages and write-back
      “The DataConnector accelerates data loading by parallelizing the reads across multiple compute nodes.”
      ↩︎ Storing training and inference data: tables, stages and write-back
      “DataSource APIs: Load data from various file formats (CSV, Parquet, images, and more) from Snowflake stages.”
      ↩︎ Storing training and inference data: tables, stages and write-back
      “Unstructured | Other Staged Files | DataSource API | N/A”
      ↩︎ Exam trap 4
    5. 6.
      “Both internal (i.e. Snowflake) and external (Amazon S3, Google Cloud Storage, or Microsoft Azure) stage references can include a path”
      ↩︎ Storing training and inference data: tables, stages and write-back
      “narrow the path to the most granular level that includes your data for improved data load performance”
      ↩︎ Storing training and inference data: tables, stages and write-back

    Ready to test yourself?

    Practise the 15 questions on this subdomain.

    Spotted a mistake, or was something unclear? Tell us.