CertSafari
    Databricks Certified Associate Developer for Apache Spark· Lessons

    Domain 7 · Lesson 32/32

    Pandas UDFs in PySpark: create and invoke pandas_udf

    Create and invoke Pandas UDF.

    17 min read
    2.5% of exam
    5 sources
    Published 21 Oct 2026
    Docs as of 30 Sep 2026

    What you will be able to do

    • Define a pandas UDF with @pandas_udf and a return type given as a DataType or a DDL string
    • Pick the pandas UDF variant from its Python type hints: Series to Series, iterator, or Series to scalar
    • Use an iterator pandas UDF to load expensive state, such as a model, once rather than per batch
    • Call a Series-to-scalar pandas UDF from groupby().agg(), select and window expressions
    • Register a pandas UDF so that it can be called from Spark SQL

    Key concept

    Pandas UDF (vectorized UDF) — A Python function that Spark calls once per batch of rows instead of once per row. Arrow moves each batch into pandas objects, and the function's type hints tell Spark which kind of UDF it is: a column transform, an iterator over batches, or an aggregate.

    1.Why pandas UDFs exist and how to declare one

    A pandas UDF runs your Python logic on whole batches of rows, not on one row at a time. An ordinary Python UDF (see row-at-a-time Python UDFs) calls the function once for every row. A pandas UDF gets a pandas Series that holds many rows, so a line like s * 0.9 is vectorized pandas code. Apache Arrow carries the data between the JVM and the Python worker, and the speed-up over row-at-a-time UDFs can reach 100x.

    Spark processes the column one batch at a time. It splits the data into batches of rows, calls your function on each batch, and concatenates the results into the output column. You never pass the whole column in yourself, and you never pass a DataFrame. You call the UDF on column expressions, like any built-in function in pyspark.sql.functions.

    There are two ways to declare one. You can put @pandas_udf(...) on the function as a decorator, or you can wrap an existing function with pandas_udf(f, returnType=...). The return type can be a DataType object such as DoubleType() or a DDL string such as "double" or "long". A third parameter, functionType, takes a PandasUDFType enum value and defaults to SCALAR. It exists only for compatibility with code written before Spark 3.0. Current code leaves it out and lets the type hints choose the variant.

    pandas_udf parameters
    ParameterAcceptsRole
    fa Python function (optional)The function to wrap, when pandas_udf is used as a wrapper rather than a decorator
    returnTypepyspark.sql.types.DataType or a DDL-formatted stringSpark type of the values the UDF produces
    functionTypea PandasUDFType enum value, default SCALARLegacy way to choose the variant; type hints are encouraged instead

    Checkpoint 1 of 9· Check yourself

    You are writing a new Spark 3.x job and need a vectorized UDF that applies a 10% discount to a price column, one batch of rows at a time. Which declaration is the recommended way to write it?

    Sources123

    2.Series to Series: vectorizing a column transform

    A Series-to-Series UDF is the pandas version of a scalar function. Its type hints are pd.Series, ... -> pd.Series: one pandas Series in for each input column, one Series out. The output must have the same length as the input, because each output row lines up with an input row. You call it in select or withColumn. For a column such as discounted_price, that means withColumn("discounted_price", apply_discount(orders.price)).

    Because each call sees only one batch, a Series-to-Series function has to be row-local. Multiplying by 0.9 or upper-casing a string is safe. Dividing by s.sum() is not, because it divides by the sum of one batch, not of the whole column. The underlying function is plain pandas code, so you can test it on a local pd.Series before Spark ever runs it, as the example below shows.

    Wrapping a plain function with pandas_udf, then testing the raw function on local pandas datapython
    # Declare the function and create the UDF
    def multiply_func(a: pd.Series, b: pd.Series) -> pd.Series:
        return a * b
    
    multiply = pandas_udf(multiply_func, returnType=LongType())
    
    # The function for a pandas_udf should be able to execute with local pandas data
    x = pd.Series([1, 2, 3])
    print(multiply_func(x, x))

    On the Spark side you invoke it as df.select(multiply(col("x"), col("x"))). Each argument is a column, given either as a Column or as a column-name string such as to_upper("name"). When the input or output column is a struct, the hint for that position is pd.DataFrame instead of pd.Series. For example, a function that splits names into first and last returns pd.DataFrame and declares the return type "first string, last string".

    Spark checks your declared returnType against what the function actually returns, but not strictly. If the returned Series has a different dtype, Spark may convert it, and that conversion is not guaranteed to be correct. Declare a long column with "long", and a float result with "double". Also, pandas UDFs do not support conditional expressions or short-circuiting in boolean expressions. A when() around the call does not stop the UDF from seeing rows it might fail on. Put the condition inside the function instead.

    Checkpoint 2 of 9· Fill the gap

    This UDF multiplies two long columns and returns one long value per input row. Which return type hint completes the function signature?

    # Declare the function and create the UDF
    def multiply_func(a: pd.Series, b: pd.Series) ->  ? :
        return a * b
    
    multiply = pandas_udf(multiply_func, returnType=LongType())
    
    # The function for a pandas_udf should be able to execute with local pandas data
    x = pd.Series([1, 2, 3])
    print(multiply_func(x, x))

    Sources13

    3.Iterator UDFs: one-time setup across many batches

    An iterator UDF receives all of its batches through a single function call. Its hints are Iterator[pd.Series] -> Iterator[pd.Series]. The function loops over the incoming batches and yields one output batch for each. Code placed before the loop therefore runs once for the whole iterator, not once per batch. That is the reason this variant exists: expensive setup such as loading a machine learning model file happens once, and inference then runs on every batch with the same model.

    The model load goes above the for loop. If you move it inside the loop, it reloads for every batch, which costs the same as loading it in the body of a plain Series-to-Series UDF on every call. Either way, the one-time setup is lost. Use try/finally or a context manager around the loop so that resources are released when the iterator is exhausted.

    Iterator of Series to Iterator of Series, with state captured before the loop and cleanup in finallypython
    y = 1  # value captured by the UDF closure
    
    @pandas_udf("long")
    def plus_y(batch_iter: Iterator[pd.Series]) -> Iterator[pd.Series]:
        try:
            for x in batch_iter:
                yield x + y
        finally:
            pass  # release resources here, if any
    
    df.select(plus_y(col("x"))).show()

    Two rules carry over from Series to Series, in a looser form. First, the length must be preserved overall, not batch by batch: the entire output of the iterator must have the same length as the entire input. This lets the function prefetch input batches. Second, the single-Series form takes exactly one Spark column. For several input columns, use the multiple-Series form, Iterator[Tuple[pd.Series, ...]] -> Iterator[pd.Series]. Its loop unpacks one tuple per batch (for a, b in iterator), and you call it with as many columns as the tuple has elements.

    Checkpoint 3 of 9· Check yourself

    A fraud-scoring job has to load a large model file and score two columns, amount and merchant_risk, as a new column. Which pandas UDF loads the model only once while getting both columns into each batch?

    Sources1

    4.Series to scalar: pandas UDFs as aggregates

    A Series-to-scalar UDF reduces one or more Series to a single value, the way mean or sum does. Only the return hint differs from Series to Series: pd.Series, ... -> float (or int, or another specific scalar type) instead of -> pd.Series. That one change turns the function into an aggregate. The declared returnType must be a primitive type, and the function may return either a Python primitive or a NumPy scalar such as numpy.float64.

    The same aggregate over the whole DataFrame and per grouppython
    @pandas_udf("double")
    def mean_udf(v: pd.Series) -> float:
        return v.mean()
    
    df.select(mean_udf(df['v'])).show()
    # +-----------+
    # |mean_udf(v)|
    # +-----------+
    # |        4.2|
    # +-----------+
    
    df.groupby("id").agg(mean_udf(df['v'])).show()
    # +---+-----------+
    # | id|mean_udf(v)|
    # +---+-----------+
    # |  1|        1.5|
    # |  2|        6.0|
    # +---+-----------+

    Where you call it decides what the aggregate is computed over. In select it covers the whole DataFrame and returns one row. In groupby("store_id").agg(...) it returns one row per group. To put the aggregate on every row, use withColumn("mean_v", mean_udf(df['v']).over(w)), where w is a Window.partitionBy('id') specification. With an ordered frame such as rowsBetween(-1, 0), the same UDF gives a moving mean. Inside window calls Spark does not copy the input Series, so the function must not mutate it or rely on its index.

    These aggregates cost memory. Unlike Spark's built-in aggregates, they do not support partial aggregation, so every value for a group is loaded into memory before the function runs. A single skewed store_id can therefore run out of memory where avg would not. Recent PySpark releases add an iterator-to-scalar form, Iterator[pd.Series] -> float, which reads a group's batches one at a time, for example by keeping a running sum and count. Only one such UDF is supported per aggregation.

    Type hints decide the variant
    Type hintsVariantTypical call site
    pd.Series, ... -> pd.SeriesSeries to Seriesselect, withColumn
    Iterator[pd.Series] -> Iterator[pd.Series]Iterator of Series to Iterator of Series (one input column)select, withColumn, with one-time setup
    Iterator[Tuple[pd.Series, ...]] -> Iterator[pd.Series]Iterator of multiple Series to Iterator of Seriesselect, withColumn with several input columns
    pd.Series, ... -> float (a scalar)Series to scalar (aggregate)select, groupBy.agg, Window via .over()
    Iterator[pd.Series] -> floatIterator of Series to scalargroupby().agg, one UDF per aggregation

    Checkpoint 4 of 9· Match them up

    Match each type-hint signature to the kind of pandas UDF it declares

    Tap a term, then the definition that fits it.

    Checkpoint 5 of 9· Check yourself

    One store_id holds most of the rows in a sales table. groupby("store_id").agg(mean_udf(df['v'])), with mean_udf typed v: pd.Series -> float, runs out of executor memory, but the same query using built-in avg succeeds. What explains the difference?

    Checkpoint 6 of 9· Exam question

    A data engineer wants to add a discounted_price column to a large Spark DataFrame orders by multiplying every value in the price column by 0.9, using a vectorized pandas UDF that operates on the whole column at once via Apache Arrow rather than row by row. Which implementation correctly achieves this?

    Checkpoint 7 of 9· Exam question

    A data scientist needs to apply a machine learning model that is expensive to load to every row of a features column across a very large DataFrame. To avoid reloading the model once per row, which pandas UDF signature should be used so the model can be loaded once per Python worker and reused across many batches?

    Checkpoint 8 of 9· Exam question

    An analyst wants to compute a weighted average price per store_id using a custom pandas UDF and Spark's grouped aggregation, so the UDF should take a Series of values from a group and return a single number for that group. Which definition and call correctly implements this?

    Sources13

    5.Calling pandas UDFs from SQL, and where mapInPandas differs

    To call a pandas UDF from SQL, register it under a name with spark.udf.register. Registration accepts plain Python functions, row-at-a-time UDFs and vectorized ones alike. Once registered, a Series-to-Series UDF can be used in any SQL expression. A Series-to-scalar UDF works in SQL as a grouped aggregate, for example SELECT sum_udf(v1) ... GROUP BY v2. This works over temporary views as well as tables.

    Registering a Series-to-Series pandas UDF and invoking it in Spark SQLpython
    @pandas_udf("integer")
    def add_one(s: pd.Series) -> pd.Series:
        return s + 1
    
    spark.udf.register("add_one", add_one)
    spark.sql("SELECT add_one(id) FROM range(3)").collect()
    # [Row(add_one(id)=1), Row(add_one(id)=2), Row(add_one(id)=3)]

    register has its own optional returnType argument, but it applies only to plain Python functions. A pandas UDF already carries the type it was declared with in @pandas_udf("integer"), so you register it with just the name and the UDF.

    DataFrame.mapInPandas() looks similar but is a different API. It is not a column function: its function takes an iterator of pandas.DataFrame batches, yields pandas.DataFrame batches, and receives an output schema. Unlike Series to Series, it may return a different number of rows than it receives, so it can filter or expand rows. A pandas UDF cannot. Grouped applyInPandas likewise works on whole pandas DataFrames per group, not on columns.

    Checkpoint 9 of 9· Fill the gap

    To make the vectorized add_one callable by name inside a Spark SQL query, which method on spark.udf completes this sample?

    @pandas_udf("integer")
    def add_one(s: pd.Series) -> pd.Series:
        return s + 1
    
    spark.udf. ? ("add_one", add_one)
    spark.sql("SELECT add_one(id) FROM range(3)").collect()
    # [Row(add_one(id)=1), Row(add_one(id)=2), Row(add_one(id)=3)]

    Sources45

    Sources

    (5)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “pandas UDFs allow vectorized operations that can increase performance up to 100x compared to row-at-a-time Python UDFs.”
      ↩︎ Why pandas UDFs exist and how to declare one
      “The Python function must accept a pandas Series as an input and return a pandas Series of the same length.”
      ↩︎ Series to Series: vectorizing a column transform
      “The length of the entire output in the iterator should be the same as the length of the entire input.”
      ↩︎ Iterator UDFs: one-time setup across many batches
      “You specify the type hints as Iterator[Tuple[pandas.Series, ...]] -> Iterator[pandas.Series].”
      ↩︎ Iterator UDFs: one-time setup across many batches
      “You use a Series to scalar pandas UDF with APIs such as select, withColumn, groupBy.agg, and pyspark.sql.Window.”
      ↩︎ Series to scalar: pandas UDFs as aggregates
      “uses Apache Arrow to transfer data and pandas to work with the data”
      ↩︎ Key concept
      “pandas UDFs allow vectorized operations that can increase performance up to 100x compared to row-at-a-time Python UDFs.”
      ↩︎ Exam trap 1
      “useful when the UDF execution requires initializing some state, for example, loading a machine learning model file”
      ↩︎ Exam trap 2
      “This type of UDF does not support partial aggregation and all data for each group is loaded into memory.”
      ↩︎ Exam trap 3
      “Spark runs a pandas UDF by splitting the data into batches of rows, calling the function for each batch, and then concatenating the results.”
      ↩︎ Exam trap 4
      “In the UDF, you can initialize some state before processing batches.”
      ↩︎ Checkpoint
      “This type of UDF does not support partial aggregation and all data for each group is loaded into memory.”
      ↩︎ Checkpoint
    2. 2.
      “This parameter exists for compatibility. Using Python type hints is encouraged.”
      ↩︎ Why pandas UDFs exist and how to declare one
    3. 3.
      “specifying pandas UDF type via functionType which will be deprecated in the future releases”
      ↩︎ Why pandas UDFs exist and how to declare one
      “When there is mismatch between them, Spark might do conversion on returned data. The conversion is not guaranteed to be correct”
      ↩︎ Series to Series: vectorizing a column transform
      “The user-defined functions do not support conditional expressions or short circuiting in boolean expressions”
      ↩︎ Series to Series: vectorizing a column transform
      “pandas.DataFrame should be used for its input or output type hint instead when the input or output column is of pyspark.sql.types.StructType”
      ↩︎ Series to Series: vectorizing a column transform
      “mutating the input series is not allowed and will cause incorrect results.”
      ↩︎ Series to scalar: pandas UDFs as aggregates
      “which is more memory-efficient than loading all data at once”
      ↩︎ Series to scalar: pandas UDFs as aggregates
      “It is preferred to specify type hints for the pandas UDF instead of specifying pandas UDF type via functionType which will be deprecated”
      ↩︎ Checkpoint
      “The length of the input is not that of the whole input column, but is the length of an internal batch”
      ↩︎ Prediction
      “Series to Scalarpandas.Series, … -> Any The function takes pandas.Series and returns a scalar value.”
      ↩︎ Checkpoint
    4. 4.
      “The user-defined function can be either row-at-a-time or vectorized.”
      ↩︎ Calling pandas UDFs from SQL, and where mapInPandas differs
      “Only valid when f is a plain Python function, not when f is already a user-defined function.”
      ↩︎ Calling pandas UDFs from SQL, and where mapInPandas differs
    5. 5.
      “It can return output of arbitrary length in contrast to some pandas UDFs such as Series to Series.”
      ↩︎ Calling pandas UDFs from SQL, and where mapInPandas differs

    Ready to test yourself?

    Practise the 11 questions on this subdomain.

    Spotted a mistake, or was something unclear? Tell us.