What you will be able to do
- Define a pandas UDF with @pandas_udf and a return type given as a DataType or a DDL string
- Pick the pandas UDF variant from its Python type hints: Series to Series, iterator, or Series to scalar
- Use an iterator pandas UDF to load expensive state, such as a model, once rather than per batch
- Call a Series-to-scalar pandas UDF from groupby().agg(), select and window expressions
- Register a pandas UDF so that it can be called from Spark SQL
Key concept
Pandas UDF (vectorized UDF) — A Python function that Spark calls once per batch of rows instead of once per row. Arrow moves each batch into pandas objects, and the function's type hints tell Spark which kind of UDF it is: a column transform, an iterator over batches, or an aggregate.
1.Why pandas UDFs exist and how to declare one
A pandas UDF runs your Python logic on whole batches of rows, not on one row at a time. An ordinary Python UDF (see row-at-a-time Python UDFs) calls the function once for every row. A pandas UDF gets a pandas Series that holds many rows, so a line like s * 0.9 is vectorized pandas code. Apache Arrow carries the data between the JVM and the Python worker, and the speed-up over row-at-a-time UDFs can reach 100x.
Spark processes the column one batch at a time. It splits the data into batches of rows, calls your function on each batch, and concatenates the results into the output column. You never pass the whole column in yourself, and you never pass a DataFrame. You call the UDF on column expressions, like any built-in function in pyspark.sql.functions.
There are two ways to declare one. You can put @pandas_udf(...) on the function as a decorator, or you can wrap an existing function with pandas_udf(f, returnType=...). The return type can be a DataType object such as DoubleType() or a DDL string such as "double" or "long". A third parameter, functionType, takes a PandasUDFType enum value and defaults to SCALAR. It exists only for compatibility with code written before Spark 3.0. Current code leaves it out and lets the type hints choose the variant.
| Parameter | Accepts | Role |
|---|---|---|
| f | a Python function (optional) | The function to wrap, when pandas_udf is used as a wrapper rather than a decorator |
| returnType | pyspark.sql.types.DataType or a DDL-formatted string | Spark type of the values the UDF produces |
| functionType | a PandasUDFType enum value, default SCALAR | Legacy way to choose the variant; type hints are encouraged instead |
Checkpoint 1 of 9· Check yourself
You are writing a new Spark 3.x job and need a vectorized UDF that applies a 10% discount to a price column, one batch of rows at a time. Which declaration is the recommended way to write it?
Current code uses pandas_udf with a return type and lets the pd.Series -> pd.Series type hints pick the Series-to-Series variant. Adding PandasUDFType.SCALAR is the legacy pre-3.0 form that is slated for deprecation. A plain @udf still runs once per row. You call the UDF on column expressions, not on a whole DataFrame.
“It is preferred to specify type hints for the pandas UDF instead of specifying pandas UDF type via functionType which will be deprecated”Source: spark.apache.org
2.Series to Series: vectorizing a column transform
A Series-to-Series UDF is the pandas version of a scalar function. Its type hints are pd.Series, ... -> pd.Series: one pandas Series in for each input column, one Series out. The output must have the same length as the input, because each output row lines up with an input row. You call it in select or withColumn. For a column such as discounted_price, that means withColumn("discounted_price", apply_discount(orders.price)).
Because each call sees only one batch, a Series-to-Series function has to be row-local. Multiplying by 0.9 or upper-casing a string is safe. Dividing by s.sum() is not, because it divides by the sum of one batch, not of the whole column. The underlying function is plain pandas code, so you can test it on a local pd.Series before Spark ever runs it, as the example below shows.
# Declare the function and create the UDF
def multiply_func(a: pd.Series, b: pd.Series) -> pd.Series:
return a * b
multiply = pandas_udf(multiply_func, returnType=LongType())
# The function for a pandas_udf should be able to execute with local pandas data
x = pd.Series([1, 2, 3])
print(multiply_func(x, x))On the Spark side you invoke it as df.select(multiply(col("x"), col("x"))). Each argument is a column, given either as a Column or as a column-name string such as to_upper("name"). When the input or output column is a struct, the hint for that position is pd.DataFrame instead of pd.Series. For example, a function that splits names into first and last returns pd.DataFrame and declares the return type "first string, last string".
Spark checks your declared returnType against what the function actually returns, but not strictly. If the returned Series has a different dtype, Spark may convert it, and that conversion is not guaranteed to be correct. Declare a long column with "long", and a float result with "double". Also, pandas UDFs do not support conditional expressions or short-circuiting in boolean expressions. A when() around the call does not stop the UDF from seeing rows it might fail on. Put the condition inside the function instead.
Checkpoint 2 of 9· Fill the gap
This UDF multiplies two long columns and returns one long value per input row. Which return type hint completes the function signature?
# Declare the function and create the UDF
def multiply_func(a: pd.Series, b: pd.Series) -> ? :
return a * b
multiply = pandas_udf(multiply_func, returnType=LongType())
# The function for a pandas_udf should be able to execute with local pandas data
x = pd.Series([1, 2, 3])
print(multiply_func(x, x))A Series-to-Series UDF is declared with pd.Series, ... -> pd.Series and returns a Series as long as its input. pd.DataFrame is only for struct input or output columns, and a scalar hint such as float would not give one output value per input row.
Source: docs.databricks.com3.Iterator UDFs: one-time setup across many batches
An iterator UDF receives all of its batches through a single function call. Its hints are Iterator[pd.Series] -> Iterator[pd.Series]. The function loops over the incoming batches and yields one output batch for each. Code placed before the loop therefore runs once for the whole iterator, not once per batch. That is the reason this variant exists: expensive setup such as loading a machine learning model file happens once, and inference then runs on every batch with the same model.
The model load goes above the for loop. If you move it inside the loop, it reloads for every batch, which costs the same as loading it in the body of a plain Series-to-Series UDF on every call. Either way, the one-time setup is lost. Use try/finally or a context manager around the loop so that resources are released when the iterator is exhausted.
y = 1 # value captured by the UDF closure
@pandas_udf("long")
def plus_y(batch_iter: Iterator[pd.Series]) -> Iterator[pd.Series]:
try:
for x in batch_iter:
yield x + y
finally:
pass # release resources here, if any
df.select(plus_y(col("x"))).show()Two rules carry over from Series to Series, in a looser form. First, the length must be preserved overall, not batch by batch: the entire output of the iterator must have the same length as the entire input. This lets the function prefetch input batches. Second, the single-Series form takes exactly one Spark column. For several input columns, use the multiple-Series form, Iterator[Tuple[pd.Series, ...]] -> Iterator[pd.Series]. Its loop unpacks one tuple per batch (for a, b in iterator), and you call it with as many columns as the tuple has elements.
Checkpoint 3 of 9· Check yourself
A fraud-scoring job has to load a large model file and score two columns, amount and merchant_risk, as a new column. Which pandas UDF loads the model only once while getting both columns into each batch?
Code placed before the loop runs once for the whole iterator, and passing several columns needs the Iterator[Tuple[pd.Series, ...]] form. The single-Series iterator form accepts only one column. Putting the load inside the loop, or in the body of a Series-to-Series UDF, reloads the model for every batch.
“In the UDF, you can initialize some state before processing batches.”Source: docs.databricks.com
Sources1
4.Series to scalar: pandas UDFs as aggregates
A Series-to-scalar UDF reduces one or more Series to a single value, the way mean or sum does. Only the return hint differs from Series to Series: pd.Series, ... -> float (or int, or another specific scalar type) instead of -> pd.Series. That one change turns the function into an aggregate. The declared returnType must be a primitive type, and the function may return either a Python primitive or a NumPy scalar such as numpy.float64.
@pandas_udf("double")
def mean_udf(v: pd.Series) -> float:
return v.mean()
df.select(mean_udf(df['v'])).show()
# +-----------+
# |mean_udf(v)|
# +-----------+
# | 4.2|
# +-----------+
df.groupby("id").agg(mean_udf(df['v'])).show()
# +---+-----------+
# | id|mean_udf(v)|
# +---+-----------+
# | 1| 1.5|
# | 2| 6.0|
# +---+-----------+Where you call it decides what the aggregate is computed over. In select it covers the whole DataFrame and returns one row. In groupby("store_id").agg(...) it returns one row per group. To put the aggregate on every row, use withColumn("mean_v", mean_udf(df['v']).over(w)), where w is a Window.partitionBy('id') specification. With an ordered frame such as rowsBetween(-1, 0), the same UDF gives a moving mean. Inside window calls Spark does not copy the input Series, so the function must not mutate it or rely on its index.
These aggregates cost memory. Unlike Spark's built-in aggregates, they do not support partial aggregation, so every value for a group is loaded into memory before the function runs. A single skewed store_id can therefore run out of memory where avg would not. Recent PySpark releases add an iterator-to-scalar form, Iterator[pd.Series] -> float, which reads a group's batches one at a time, for example by keeping a running sum and count. Only one such UDF is supported per aggregation.
| Type hints | Variant | Typical call site |
|---|---|---|
| pd.Series, ... -> pd.Series | Series to Series | select, withColumn |
| Iterator[pd.Series] -> Iterator[pd.Series] | Iterator of Series to Iterator of Series (one input column) | select, withColumn, with one-time setup |
| Iterator[Tuple[pd.Series, ...]] -> Iterator[pd.Series] | Iterator of multiple Series to Iterator of Series | select, withColumn with several input columns |
| pd.Series, ... -> float (a scalar) | Series to scalar (aggregate) | select, groupBy.agg, Window via .over() |
| Iterator[pd.Series] -> float | Iterator of Series to scalar | groupby().agg, one UDF per aggregation |
The -> pd.Series hint makes it a Series-to-Series UDF. It returns one value per input row, not one per group, so it is not an aggregate. Its sum() is also taken over one Arrow batch, not over a store. A per-group number needs a scalar return hint such as -> float.
Checkpoint 4 of 9· Match them up
Match each type-hint signature to the kind of pandas UDF it declares
Tap a term, then the definition that fits it.
pandas_udf picks the variant from the type hints. A scalar return hint such as -> float turns a function into an aggregate, and an Iterator input with a scalar return gives the batch-by-batch aggregate form.
“Series to Scalarpandas.Series, … -> Any The function takes pandas.Series and returns a scalar value.”Source: spark.apache.org
Checkpoint 5 of 9· Check yourself
One store_id holds most of the rows in a sales table. groupby("store_id").agg(mean_udf(df['v'])), with mean_udf typed v: pd.Series -> float, runs out of executor memory, but the same query using built-in avg succeeds. What explains the difference?
Unlike built-in aggregates, a Series-to-scalar pandas UDF gets each group as a single Series, so a skewed group has to fit in memory all at once. The Iterator[pd.Series] -> float form avoids this by reading the group one batch at a time.
“This type of UDF does not support partial aggregation and all data for each group is loaded into memory.”Source: docs.databricks.com
Checkpoint 6 of 9· Exam question
A data engineer wants to add a discounted_price column to a large Spark DataFrame orders by multiplying every value in the price column by 0.9, using a vectorized pandas UDF that operates on the whole column at once via Apache Arrow rather than row by row. Which implementation correctly achieves this?
Correct answer: A — Decorate a function typed s: pd.Series -> pd.Series with @pandas_udf(DoubleType()), then call it inside withColumn("discounted_price", apply_discount(orders.price)) so Arrow transfers the whole column in batches.
- A. This is correct because pandas_udf with a Series-to-Series signature receives the full price column as a pandas Series through Arrow, multiplies it in one vectorized operation, and returns a same-length Series that withColumn can attach directly to the DataFrame.
- B. The plain udf decorator does not use Arrow batching, so even though the function body is identical, Spark serializes and deserializes each row individually with Python pickle, losing the vectorized performance benefit pandas_udf is meant to provide.
- C. Returning a scalar float from
s.mean() * 0.9makes this a grouped-aggregate pandas UDF, which is meant to be called inside groupby().agg() to produce one value per group, not to multiply every row's own price and attach it as a new column. - D. pandas_udf-wrapped functions are called like any other Spark column function, so passing the DataFrame and a column name string as two arguments produces a TypeError instead of the expected Column object needed inside withColumn.
- E. Combining the deprecated PandasUDFType.SCALAR argument with modern type-hint-based pandas_udf syntax mixes two incompatible calling conventions; PandasUDFType has been removed in current Spark releases in favor of inferring the UDF type entirely from the function's type hints, so this call raises an error.
Checkpoint 7 of 9· Exam question
A data scientist needs to apply a machine learning model that is expensive to load to every row of a features column across a very large DataFrame. To avoid reloading the model once per row, which pandas UDF signature should be used so the model can be loaded once per Python worker and reused across many batches?
Correct answer: A — Type the function as Iterator[pd.Series] -> Iterator[pd.Series], load the model once before the loop over the iterator, and yield predictions for each incoming batch of the features column inside the loop.
- A. The iterator-to-iterator signature runs the function once per worker process, so loading the model before entering the for-loop over the iterator means the model is initialized a single time and then reused to score every batch of the features column that arrives, avoiding repeated load costs.
- B. The scalar Series-to-Series signature is invoked once per batch by Spark, so loading the model inside the function body still reloads it for every batch rather than once per worker, which does not solve the expensive load problem described.
- C. A Series-to-scalar signature is a grouped-aggregate pandas UDF meant to collapse each group into one value inside groupby().agg(); it does not apply the model row-by-row across the whole features column the way this scenario requires.
- D. Using the iterator signature but reloading the model inside the loop still recreates the model on every batch instead of once per worker, defeating the purpose of choosing the iterator-based signature in the first place.
- E. PandasUDFType has been removed in current Spark releases, so pairing it with modern type-hint-based pandas_udf syntax is not a supported calling convention and this definition raises an error rather than running successfully.
Checkpoint 8 of 9· Exam question
An analyst wants to compute a weighted average price per store_id using a custom pandas UDF and Spark's grouped aggregation, so the UDF should take a Series of values from a group and return a single number for that group. Which definition and call correctly implements this?
Correct answer: A — @pandas_udf(DoubleType()) on def weighted_avg(price: pd.Series) -> float: return float(price.mean()), called as orders.groupby("store_id").agg(weighted_avg(orders.price)).
- A. Returning a single float from a Series input is exactly the grouped-aggregate shape pandas_udf expects, so calling it inside groupby("store_id").agg(...) produces one weighted-average value per store as intended.
- B. The function itself is a valid grouped-aggregate UDF, but calling it inside withColumn on an ungrouped DataFrame is not a supported use of a Series-to-scalar pandas UDF, and Spark raises an error because no grouping context exists to collapse into a scalar.
- C. Returning a Series the same length as the input instead of a single float makes this a Series-to-Series pandas UDF, which Spark cannot use inside groupby().agg() because an aggregation call expects one scalar result per group, not one value per row.
- D. Without the pandas_udf decorator, this is treated as a plain Python function rather than a registered Spark UDF, so calling it with a Column argument inside agg() fails instead of computing a per-group average.
- E. The iterator-based signature is meant to be called directly on ungrouped batches of data, not passed to groupby().agg(), so wrapping it in a group aggregation this way does not match any supported pandas_udf calling pattern and raises an error.
5.Calling pandas UDFs from SQL, and where mapInPandas differs
To call a pandas UDF from SQL, register it under a name with spark.udf.register. Registration accepts plain Python functions, row-at-a-time UDFs and vectorized ones alike. Once registered, a Series-to-Series UDF can be used in any SQL expression. A Series-to-scalar UDF works in SQL as a grouped aggregate, for example SELECT sum_udf(v1) ... GROUP BY v2. This works over temporary views as well as tables.
@pandas_udf("integer")
def add_one(s: pd.Series) -> pd.Series:
return s + 1
spark.udf.register("add_one", add_one)
spark.sql("SELECT add_one(id) FROM range(3)").collect()
# [Row(add_one(id)=1), Row(add_one(id)=2), Row(add_one(id)=3)]register has its own optional returnType argument, but it applies only to plain Python functions. A pandas UDF already carries the type it was declared with in @pandas_udf("integer"), so you register it with just the name and the UDF.
DataFrame.mapInPandas() looks similar but is a different API. It is not a column function: its function takes an iterator of pandas.DataFrame batches, yields pandas.DataFrame batches, and receives an output schema. Unlike Series to Series, it may return a different number of rows than it receives, so it can filter or expand rows. A pandas UDF cannot. Grouped applyInPandas likewise works on whole pandas DataFrames per group, not on columns.
Checkpoint 9 of 9· Fill the gap
To make the vectorized add_one callable by name inside a Spark SQL query, which method on spark.udf completes this sample?
@pandas_udf("integer")
def add_one(s: pd.Series) -> pd.Series:
return s + 1
spark.udf. ? ("add_one", add_one)
spark.sql("SELECT add_one(id) FROM range(3)").collect()
# [Row(add_one(id)=1), Row(add_one(id)=2), Row(add_one(id)=3)]spark.udf.register gives a plain function, a row-at-a-time UDF or a pandas UDF a name that SQL statements can call. No returnType is passed here because the pandas UDF already carries the type from @pandas_udf("integer"). createOrReplaceTempView names a DataFrame, not a function, and pandas_udf creates the UDF but does not make it available to SQL.
Source: docs.databricks.comSources
(5)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/udf/pandasOfficial docs
“pandas UDFs allow vectorized operations that can increase performance up to 100x compared to row-at-a-time Python UDFs.”
↩︎ Why pandas UDFs exist and how to declare one“The Python function must accept a pandas Series as an input and return a pandas Series of the same length.”
↩︎ Series to Series: vectorizing a column transform“The length of the entire output in the iterator should be the same as the length of the entire input.”
↩︎ Iterator UDFs: one-time setup across many batches“You specify the type hints as Iterator[Tuple[pandas.Series, ...]] -> Iterator[pandas.Series].”
↩︎ Iterator UDFs: one-time setup across many batches“You use a Series to scalar pandas UDF with APIs such as select, withColumn, groupBy.agg, and pyspark.sql.Window.”
↩︎ Series to scalar: pandas UDFs as aggregates“uses Apache Arrow to transfer data and pandas to work with the data”
↩︎ Key concept“pandas UDFs allow vectorized operations that can increase performance up to 100x compared to row-at-a-time Python UDFs.”
↩︎ Exam trap 1“useful when the UDF execution requires initializing some state, for example, loading a machine learning model file”
↩︎ Exam trap 2“This type of UDF does not support partial aggregation and all data for each group is loaded into memory.”
↩︎ Exam trap 3“Spark runs a pandas UDF by splitting the data into batches of rows, calling the function for each batch, and then concatenating the results.”
↩︎ Exam trap 4“In the UDF, you can initialize some state before processing batches.”
↩︎ Checkpoint“This type of UDF does not support partial aggregation and all data for each group is loaded into memory.”
↩︎ Checkpoint - 2.
“This parameter exists for compatibility. Using Python type hints is encouraged.”
↩︎ Why pandas UDFs exist and how to declare one - 3.https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.pandas_udf.htmlSecondary source
“specifying pandas UDF type via functionType which will be deprecated in the future releases”
↩︎ Why pandas UDFs exist and how to declare one“When there is mismatch between them, Spark might do conversion on returned data. The conversion is not guaranteed to be correct”
↩︎ Series to Series: vectorizing a column transform“The user-defined functions do not support conditional expressions or short circuiting in boolean expressions”
↩︎ Series to Series: vectorizing a column transform“pandas.DataFrame should be used for its input or output type hint instead when the input or output column is of pyspark.sql.types.StructType”
↩︎ Series to Series: vectorizing a column transform“mutating the input series is not allowed and will cause incorrect results.”
↩︎ Series to scalar: pandas UDFs as aggregates“which is more memory-efficient than loading all data at once”
↩︎ Series to scalar: pandas UDFs as aggregates“It is preferred to specify type hints for the pandas UDF instead of specifying pandas UDF type via functionType which will be deprecated”
↩︎ Checkpoint“The length of the input is not that of the whole input column, but is the length of an internal batch”
↩︎ Prediction“Series to Scalarpandas.Series, … -> Any The function takes pandas.Series and returns a scalar value.”
↩︎ Checkpoint - 4.
“The user-defined function can be either row-at-a-time or vectorized.”
↩︎ Calling pandas UDFs from SQL, and where mapInPandas differs“Only valid when f is a plain Python function, not when f is already a user-defined function.”
↩︎ Calling pandas UDFs from SQL, and where mapInPandas differs - 5.
“It can return output of arbitrary length in contrast to some pandas UDFs such as Series to Series.”
↩︎ Calling pandas UDFs from SQL, and where mapInPandas differs