Feature Engineering with Snowpark DataFrames and Snowflake ML Preprocessing

1.Snowpark DataFrames and pandas on Snowflake

Almost all feature engineering in Snowflake starts with a DataFrame. A Snowpark DataFrame works more like a query than a container of rows. You build it in three steps: construct it from a source (a table, a staged file, local values or a SQL statement), describe the transformations (select, filter, sort, group), and then call an action such as collect() to execute it. Until you call that action, nothing runs.

If your team already writes pandas, pandas on Snowflake (the Snowpark pandas API, built on Modin) lets you keep that syntax. You install snowflake-snowpark-python[modin] and change your imports. For large data, it translates your pandas operations into SQL and runs them in Snowflake.

import modin.pandas as pd
import snowflake.snowpark.modin.plugin

From Snowpark Python 1.40.0, hybrid execution is on by default. pandas on Snowflake decides for itself whether an operation runs locally or in Snowflake, and df.get_backend() tells you which one it picked. A 10-million-row table read with pd.read_snowflake stays in Snowflake. A filtered aggregation that returns 7 rows can move to local pandas. Whichever backend runs it, the object is still a modin.pandas DataFrame, so code further down the pipeline doesn't need to change.

# Create a Snowpark pandas DataFrame from existing Snowflake table
df = pd. ? ('SNOWFALL')

Sources1

2.Derived features: aggregates such as average spend

Many columns can be used as features just as they are. Others become more useful once you derive something from them. Snowflake's own example is turning a timestamp into a day-of-week feature so a model can pick up weekly patterns. The documentation also names aggregating, differentiating and time-shifting as common transformations. A per-customer average spend is an aggregate of this kind: you group the transactions by the entity and take the mean of the amount.

In pandas on Snowflake you write that as an ordinary groupby. In the documentation's example, it computes the average snowfall for each location in a single expression. Because read_snowflake also accepts a SQL query, you can write the same aggregate in SQL and get a DataFrame back.

summary_df = pd.read_snowflake("SELECT LOCATION, AVG(SNOWFALL) AS avg_snowfall FROM SNOWFALL GROUP BY LOCATION")

Sources2

3.Choosing where preprocessing runs

Once you have candidate features, preprocessing makes them ready for a model: scaling numeric columns and encoding categorical ones. Snowflake ML offers three ways to do this, and the right one depends mainly on data size and on how much custom logic you need.

The Snowflake ML preprocessors are a deliberate subset of scikit-learn's preprocessors that covers the most common cases. In exchange for that narrower set, they push the work down to a warehouse instead of pulling the data into memory. Ray map_batches is lazy too: nothing is processed until you materialize the dataset.

Sources3

4.Scaling, normalization and one-hot encoding in a Snowflake ML Pipeline

The Snowflake ML preprocessors live in snowflake.ml.modeling.preprocessing, and they work on Snowpark DataFrames instead of local arrays. Each transformer names the columns it reads in input_cols and the columns it writes in output_cols. You chain transformers with snowflake.ml.modeling.pipeline.Pipeline, just as you would in scikit-learn. In the example below, StandardScaler scales AGE and INCOME, and OneHotEncoder turns the categorical CITY column into indicator columns.

# Define Snowflake ML preprocessors
scaler = StandardScaler(input_cols=['AGE', 'INCOME'], output_cols=['AGE_SCALED', 'INCOME_SCALED'])
encoder = OneHotEncoder(input_cols=['CITY'], output_cols=['CITY_ENCODED'])

pipeline = Pipeline(steps=[
    ('scaling', scaler),
    ('encoding', encoder)
])

# Fit and transform data in Snowflake (distributed)
result = pipeline.fit_transform(df)
result.show()

Normalization. The only normalization these sources show is standardization: subtract the mean and divide by the standard deviation. StandardScaler does that inside Snowflake. The Ray example writes the same z-score formula by hand inside a batch function. Min-max scaling and other normalizers aren't covered in the documentation provided for this lesson, so look them up in the Snowflake ML preprocessing reference.

def preprocess_batch(batch: pd.DataFrame) -> pd.DataFrame:
    batch['AGE_SCALED'] = (batch['age'] - batch['age'].mean()) / batch['age'].std()
    return batch

Sources3

5.Binarizing: binning intervals, one-hot and label encoding

Binarizing means turning values into a small set of discrete codes. For binning continuous data into intervals, Snowflake SQL provides WIDTH_BUCKET. It divides a range into buckets of equal width and returns the number of the bucket each value falls into. Values below min_value get bucket 0. Values greater than or equal to max_value get num_buckets + 1, so outliers are labelled rather than dropped. If any input is NULL, the result is NULL.

WIDTH_BUCKET( <expr> , <min_value> , <max_value> , <num_buckets> )

One-hot encoding is the other form of binarizing covered here. OneHotEncoder (shown in the pipeline in the previous section, and in scikit-learn for the OSS route) replaces a categorical column with indicator columns. Label encoding, which maps each category to a single integer code, is not covered in the documentation provided for this lesson. Check the Snowflake ML preprocessing reference before relying on a particular class for it.

Sources4

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Snowflake ML preprocessors offer every transformer that scikit-learn does, just distributed.Why is that wrong?

   They are a subset that covers the most common cases. For anything outside it, use OSS preprocessors on smaller data or custom Ray logic.

   Covered in Choosing where preprocessing runs

2. 2.For a large production table, running scikit-learn preprocessing is equivalent to using Snowflake ML preprocessors.Why is that wrong?

   scikit-learn runs in memory and suits small or medium data. Snowflake ML preprocessors push down to the warehouse and are the recommended choice for large production workloads.

   Covered in Scaling, normalization and one-hot encoding in a Snowflake ML Pipeline