What you will be able to do
- Write transformations as pure DataFrame-to-DataFrame functions, chain them with DataFrame.transform, and keep them in an importable package that pytest can test.
- Predict whether assertDataFrameEqual and assertSchemaEqual pass or fail by default, and choose the right flag (checkRowOrder, rtol/atol, ignoreNullable, the diff options) for a given test.
- Set up Spark session fixtures for unit tests that run locally, through Databricks Connect, or on a cluster.
- Tell unit tests apart from integration tests that run pipeline code or jobs, and know the isolation rules and prerequisites for pipeline tests.
Key concept
Pure DataFrame-to-DataFrame transformation — A function that takes a DataFrame (plus optional parameters), returns a new DataFrame, and does no reading or writing of tables. It can run on a few hand-built rows in a test, and the same function can be chained into the production pipeline with .transform.
1.Where testable transformation code lives
A transformation hidden inside a notebook cell can only be checked by running the whole notebook against whatever data it happens to read. Unit testing turns that around. You test each self-contained function on a handful of fake rows that you control, before it ever touches a real table. Writing to production from a test is the obvious risk, and it is worst for functions that add, remove or change data. The safe pattern is fake data that looks like production but is not production.
For Python, Databricks recommends keeping functions and their tests in files outside notebooks, because test frameworks are built to run tests that way. Put the transformation functions in the importable src package of your project (the layout comes from Modular Python project layout for Declarative Automation Bundles (formerly DABs)). Then a test file can import a function and call it directly, without running a notebook. The notebook becomes a thin caller of the same package. Testing the shared code by itself also catches a broken function before the notebook that depends on it fails later in a scheduled run.
├── covid_analysis
│ └── transforms.py
├── notebooks
│ ├── covid_eda_modular
│ └── covid_eda_raw (optional)
├── requirements.txt
└── tests
├── testdata.csv
└── transforms_test.pypytest finds tests by name. It collects files that match test_*.py or *_test.py, and inside them it runs functions whose names start with test_. The workspace Tests pane uses the same rules. It shows up when a matching file is open in the editor, and you can run all tests, rerun only the failed ones, or run a single test from there. In a Git folder, pipeline or bundle space with an attached cluster, it discovers tests across every file in the folder.
Design matters as much as location. A function that is easy to test returns one predictable kind of result. A function that returns False in one case and a DataFrame in another forces every test to handle both. A transformation should always return a DataFrame, and it should raise an error when something is wrong instead of quietly returning something else.
Checkpoint 1 of 9· Check yourself
A team's PySpark cleanup function is defined in a notebook cell. They want to unit test it with pytest before it ever touches a real table. What layout does Databricks recommend for Python?
For Python, Databricks recommends keeping functions and their tests in files outside notebooks, because test frameworks are built to run tests that way. A test can then import the function and call it directly on fake rows, and the notebook calls the same package. Keeping the tests in notebooks is the recommendation for Scala, not Python.
“For Python and R notebooks, Databricks recommends storing functions and their unit tests outside of notebooks.”Source: docs.databricks.com
2.Composing steps with DataFrame.transform
DataFrame.transform(func, *args, **kwargs) passes the DataFrame to func and returns whatever DataFrame func returns. The method just chains calls, but it shapes how you write code. Every step must be a callable that takes a DataFrame and returns one, so every step is also a function you can test on its own.
from pyspark.sql import functions as sf
df = spark.createDataFrame([(1, 1.0), (2, 2.0)], ["int", "float"])
def cast_all_to_int(input_df):
return input_df.select([sf.col(c).cast("int") for c in input_df.columns])
def sort_columns_asc(input_df):
return input_df.select(*sorted(input_df.columns))
df.transform(cast_all_to_int).transform(sort_columns_asc).show()
# +-----+---+
# |float|int|
# +-----+---+
# | 1| 1|
# | 2| 2|
# +-----+---+Parameters go through .transform itself. With def add_n(input_df, n), you can write df.transform(add_n, 1).transform(add_n, n=10): extra positional and keyword arguments are passed on to the function. So a configurable step does not need a closure or a global variable, and a test can call add_n(small_df, 1) directly with the value it wants to check.
Refactoring a large ETL function with .transform follows a clear pattern. Split it into named, importable functions such as add_flag, dedupe_rows and cast_types, each returning df.withColumn(...) or something similar. Production code chains them: df.transform(add_flag).transform(dedupe_rows).transform(cast_types). A separate test module imports each function and checks its output against a small expected DataFrame. When the chain breaks, the failing test names the step. A single check at the end of the chain would only tell you that something went wrong somewhere.
Two habits work against this. First, .transform takes the function object, not its name as a string, so looking up steps by name with eval() adds risk and gains nothing. Second, a try/except that catches errors and returns the input unchanged breaks the guarantee the tests rely on. The broken step still returns a DataFrame, the chain keeps going, and the error only shows up as wrong data further down the line.
Checkpoint 2 of 9· Fill the gap
The second step in this chain should sort the columns. What do you pass to .transform so that it runs sort_columns_asc on the DataFrame?
from pyspark.sql import functions as sf
df = spark.createDataFrame([(1, 1.0), (2, 2.0)], ["int", "float"])
def cast_all_to_int(input_df):
return input_df.select([sf.col(c).cast("int") for c in input_df.columns])
def sort_columns_asc(input_df):
return input_df.select(*sorted(input_df.columns))
df.transform(cast_all_to_int).transform( ? ).show().transform takes the function object itself and calls it with the current DataFrame. A string name is not callable. Writing sort_columns_asc(df) calls the function on the original df before .transform runs, so .transform gets a DataFrame instead of a function and the cast step is skipped. Writing sort_columns_asc() fails because the function needs its input DataFrame.
Checkpoint 3 of 9· Exam question
A team is refactoring a monolithic PySpark ETL function into small, independently testable steps using DataFrame.transform. Which of the following practices align with recommended patterns for writing unit-testable transformation chains with .transform? (Select all that apply)(Select 3)
Correct answers: A, B, C — Write each transformation as a standalone function accepting and returning a DataFrame, such as def add_flag(df): return df.withColumn(...), so it can be unit tested in isolation before being composed.; Chain the pipeline as df.transform(add_flag).transform(dedupe_rows).transform(cast_types), keeping each step a named, importable function that a separate test module can exercise on its own.; Call assertDataFrameEqual against a small, hand-built expected DataFrame after each individual transformation function, rather than validating correctness only at the end of the full chain.
- A. Defining a transformation as a plain function of DataFrame in, DataFrame out is exactly what
.transformexpects, and it makes the function directly callable and assertable in isolation with a small synthetic input. This is a core pattern for unit-testable pipelines. - B. Chaining named functions through
.transformkeeps the pipeline readable while preserving each step as an independently importable unit that a test module can call directly. This composition style is what makes granular unit testing practical. - C. Testing each transformation function against a small expected DataFrame catches regressions at the step where they are introduced, rather than only at the end of a long chain where the root cause is harder to isolate. This is a standard unit-testing practice for
.transformchains. - D. Silently swallowing exceptions and returning the unmodified input defeats the purpose of testing: a broken transformation would never surface as a test failure, masking real regressions. This is an anti-pattern, not a recommended practice.
- E.
DataFrame.transformtakes a callable, not a string, and resolving function names viaeval()introduces an unnecessary code-injection risk while contradicting the actual API contract. This describes incorrect usage of.transform.
Sources4
3.assertDataFrameEqual and assertSchemaEqual defaults
pyspark.testing.assertDataFrameEqual(actual, expected) compares two DataFrames' schemas and rows and raises an assertion error when they differ. That makes it a natural fit for a pytest test. To use it, build the expected DataFrame by hand with createDataFrame, run the function under test on a small input, and compare the two. What decides whether a test passes is the set of default arguments.
Ignoring row order is usually what you want. Spark does not promise an output order after a groupBy or a shuffle, so a test that compared by position would fail at random. It is the wrong default when order is the thing you are testing, for example a step that has to keep a sequence for a later windowing calculation. In that case, call assertDataFrameEqual(actual_df, expected_df, checkRowOrder=True). The function does compare cell values, and a failure raises an error immediately, not a warning. When a test passes on reordered rows, the comparison worked as designed: the default simply ignores order.
| Argument | Default | Effect on the comparison |
|---|---|---|
| checkRowOrder | False | Rows are compared as a sorted set; pass True to fail on reordering |
| rtol / atol | 1e-05 / 1e-08 | Floats within relative/absolute tolerance count as equal |
| ignoreNullable | True | A nullable-only difference in a field does not fail the test |
| ignoreColumnOrder | False | Columns must appear in the same order |
| ignoreColumnName | False | Column names must match |
| ignoreColumnType | False | Data types must match |
| maxErrors | None | Caps how many differing rows are reported |
| showOnlyDiff / includeDiffRows | False / False | Limit the printed output to differing rows / attach the differing rows to the raised error |
Floating-point results rarely match exactly. An average computed one way might give 0.30000000000000004 where the expected value is 0.3. The rtol and atol tolerances absorb differences like that, so you do not need to round inside the test. Loosen them only when the computation really is approximate. A tolerance wide enough to hide a units bug is worse than no test.
When the assertion fails, the error shows the rows that differ between actual and expected. On a wide or long result that output gets hard to read. showOnlyDiff=True drops the rows that match, and maxErrors limits how many mismatches are reported. includeDiffRows=True attaches the differing rows to the raised error so you can inspect them in code.
Schema mismatches are the other common failure, and they often come from the expected DataFrame. createDataFrame infers Python integers as LongType and marks every inferred field as nullable. A column that the code under test produces as IntegerType will therefore fail on type. Declare the expected schema with a DDL string such as "age INT, name STRING" instead of relying on inference. Nullability works the other way: with ignoreNullable=True, an expected field that is nullable and an actual field declared False still compare equal.
To check only the schema, use assertSchemaEqual from pyspark.testing. It compares two StructType values directly, with no rows involved, and handles nullability the same relaxed way by default. Column names, column order and data types must all match exactly. Changing a field from non-nullable to nullable, and nothing else, does not fail the test unless you pass ignoreNullable=False. Use it when a step's job is to define a table's structure, such as a cast or rename step, and the row values are not what you are testing.
Checkpoint 4 of 9· Check yourself
You call assertDataFrameEqual(actual_df, expected_df) with default arguments. Which of these differences between the two DataFrames makes the test fail?
Data types are strict by default (ignoreColumnType=False), so IntegerType against an inferred LongType fails. A difference only in nullability passes because ignoreNullable=True. A tiny float difference passes because it is within the rtol/atol tolerance. Reordered rows pass because checkRowOrder=False.
“checkRowOrder=False, rtol=1e-05, atol=1e-08, ignoreNullable=True, ignoreColumnOrder=False, ignoreColumnName=False, ignoreColumnType=False”Source: spark.apache.org
Checkpoint 5 of 9· Exam question
A data engineer wrote a pytest test that calls assertDataFrameEqual(actual_df, expected_df) to validate a transformation that must preserve input row order for a downstream windowing calculation. The test passes even though a manual review shows the rows in actual_df appear in a different order than in expected_df. What causes this behavior, and how should the engineer fix the test?
Correct answer: A — assertDataFrameEqual defaults to checkRowOrder=False, so it sorts rows before comparing content and ignores their original sequence; passing checkRowOrder=True makes the assertion sensitive to row order.
- A.
assertDataFrameEqualships withcheckRowOrder=Falseby default, meaning rows are effectively compared as a multiset rather than a sequence. Explicitly passingcheckRowOrder=Trueis the documented way to make the assertion fail when row ordering changes. - B. The function does inspect every cell value, not just row counts and names, so this description of its behavior is wrong.
ignoreColumnOrdercontrols whether columns must appear in the same left-to-right position, which is unrelated to row sequence. - C. There is no cross-run caching of assertion outcomes in PySpark's testing utilities; each call independently re-evaluates both DataFrames. Clearing
.pytest_cachewould have no effect on this comparison. - D. The function raises
PySparkAssertionErrorrather than only warning when DataFrames differ, so a genuine mismatch would stop the test. RunningassertSchemaEqualseparately would confirm schema equality but would not reveal anything about row ordering or values.
Checkpoint 6 of 9· Exam question
Which statement accurately describes the default behavior of assertSchemaEqual when comparing two StructType schemas in a PySpark unit test?
Correct answer: A — By default, nullable flags are ignored while column names, order, and data types must match exactly, so relaxing a field from non-nullable to nullable alone will not fail the test unless ignoreNullable=False is passed.
- A.
assertSchemaEqualdefaults toignoreNullable=True, so nullable differences alone are tolerated, while column names, positional order, and data types are still compared and must match. This is the correct description of its out-of-the-box behavior. - B. The default is the opposite:
ignoreNullable=Truemeans nullable-only differences are tolerated, not treated as mismatches.ignoreColumnOrderalso controls positional comparison, not nullable handling, so this option misattributes both defaults. - C. Data types are compared by default, not ignored; a genuine
IntegerTypevsStringTypedifference would raise a schema mismatch.ignoreColumnTypemust be explicitly set toTrueto obtain the behavior described here. - D. Column names are compared by default, not ignored;
ignoreColumnNamemust be explicitly set toTrueto switch to purely positional, type-only comparison. Describing this as the default reverses the actual behavior.
4.Spark sessions for unit tests
A unit test needs a Spark session, and where that session runs is the main choice in setting one up. The cheapest option is local. Functions that accept and return DataFrames in local memory can be tested with plain pytest, with no workspace connection at all. That suits every function written in the .transform style above.
When the tests need Databricks compute, provide the session through a pytest fixture. A file that is not a notebook does not get the spark variable a notebook creates for you. The fixture builds the session, and each test that lists spark as a parameter receives it.
from pyspark.sql import SparkSession
import pytest
@pytest.fixture
def spark() -> SparkSession:
# Create a SparkSession (the entry point to Spark functionality) on
# the cluster in the remote Databricks workspace. Unit tests do not
# have access to this SparkSession by default.
return SparkSession.builder.getOrCreate()With Databricks Connect, the fixture calls DatabricksSession.builder.getOrCreate(), and the DataFrame work runs on a remote cluster or on serverless compute while pytest runs on your laptop. This setup has two rules that trip people up. Run from a terminal, pytest only uses the DEFAULT configuration profile, so that profile has to name the compute. And Databricks Connect cannot be installed alongside a standalone PySpark package. A project that has pyspark pinned for local tests must remove it before adding Databricks Connect.
Checkpoint 7 of 9· Fill the gap
A test file outside a notebook has no spark variable of its own. Which decorator lets every test that lists spark as a parameter receive the session this function returns?
from pyspark.sql import SparkSession
import pytest
@pytest. ?
def spark() -> SparkSession:
# Create a SparkSession (the entry point to Spark functionality) on
# the cluster in the remote Databricks workspace. Unit tests do not
# have access to this SparkSession by default.
return SparkSession.builder.getOrCreate()@pytest.fixture turns spark() into a fixture, so pytest builds the session and passes it to any test that names spark as a parameter. mark.usefixtures applies fixtures that already exist and does not define one. mark.parametrize supplies lists of argument values, not a shared session.
Checkpoint 8 of 9· Check yourself
A developer has two Databricks Connect profiles. DEFAULT names the workspace but no compute, and a profile called dev-cluster names a cluster. They run pytest from a terminal, and the fixture calls DatabricksSession.builder.getOrCreate(). Session creation fails. What fixes it?
When pytest runs from a terminal with Databricks Connect, it uses only the DEFAULT profile, so the cluster or serverless compute has to be set in that profile. A standalone PySpark package cannot be installed alongside Databricks Connect.
“When running Databricks Connect from the terminal, pytest only works with the DEFAULT configuration profile.”Source: docs.databricks.com
5.Integration tests: pipelines and jobs against real tables
An integration test runs the real wiring: table reads, pipeline flows, a deployed job. It is not limited to a pure function. For Lakeflow pipelines, a pipeline unit testing framework (in Beta) sits between the two levels. You mock the input tables by name, run part of the pipeline graph on the pipeline's own compute, and then check the output tables. Its test_spark fixture provides a SparkSession that sends table operations to a temporary test schema in the pipeline's default catalog, so tests write their outputs there and leave production tables alone.
def test_counts_full_dataframe(test_spark):
mock_users(test_spark)
status = test_pipeline.run(test_spark, set(["catalog.schema.users", "catalog.schema.counts"]))
assert status.is_success
result = test_spark.table("catalog.schema.counts")
expected = test_spark.createDataFrame(
[("admin", 2, 1), ("user", 2, 1)],
schema=["user_type", "total_count", "count_valid_emails"]
)
assertDataFrameEqual(result, expected)In this test, counts comes from a groupBy, so its row order is not fixed. The default checkRowOrder=False is what keeps the comparison stable. TestPipeline.run takes a set of table names to run. Because counts depends on users, both are listed. Calling test_pipeline.run(test_spark) with no set runs every table in the pipeline.
The framework has prerequisites that are easy to miss:
- You must be the pipeline Owner. CAN RUN and CAN MANAGE are not enough. You also need USE CATALOG and CREATE SCHEMA on the default catalog.
- The pipeline must use triggered (non-continuous) mode and run on Databricks Runtime 18.1 or above.
- Spark Connect is not supported.
Isolation only covers operations that refer to a table by its name. A read by path (/Volumes/..., s3://...) or from a connector such as Kafka or Auto Loader goes straight to the real system. So do GRANT statements, tag changes and CREATE SCHEMA. A test file that looks harmless can still run a flow that does any of these. Select only outputs whose upstream tables are all mocked by name.
At the job level, a Lakeflow Job can run the test suite as its own task and then run the main notebook, either on demand or on a schedule. Point the job at a committed Git reference, not a working copy, so it tests the code that was actually merged. Unit tests on small DataFrames catch logic errors in seconds. Pipeline and job runs catch the errors that only appear when the pieces run together on real compute.
No. The test session only redirects operations that use a table name. The Auto Loader read goes to the real storage location. Either mock that input as a named table, or leave that output out of the run.
Checkpoint 9 of 9· Fill the gap
This pipeline test compares the counts table with an expected DataFrame. counts is built from the users table. Which table reference completes the set passed to test_pipeline.run, so the test runs the right part of the graph and stays isolated from production?
def test_counts_full_dataframe(test_spark):
mock_users(test_spark)
status = test_pipeline.run(test_spark, set([ ? , "catalog.schema.counts"]))
assert status.is_success
result = test_spark.table("catalog.schema.counts")
expected = test_spark.createDataFrame(
[("admin", 2, 1), ("user", 2, 1)],
schema=["user_type", "total_count", "count_valid_emails"]
)
assertDataFrameEqual(result, expected)counts depends on users, so the test lists both tables by their normal catalog.schema.table names. The test SparkSession sends named-table operations to the temporary test schema itself, so you don't write the test schema name. A path reference skips that isolation and reaches the real storage. Listing counts twice would leave out the users table it depends on.
Source: docs.databricks.comSources
(9)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/notebooks/testingOfficial docs
“it is a best practice to not run unit tests against functions that work with data in production”
↩︎ Where testable transformation code lives“For Python and R notebooks, Databricks recommends storing functions and their unit tests outside of notebooks.”
↩︎ Where testable transformation code lives“a function should return a single predictable outcome and be of a single data type”
↩︎ Where testable transformation code lives“Because this file is not a Databricks notebook, you”
↩︎ Spark sessions for unit tests - 2.
“You want to catch failures in your shared code first before having your main notebook eventually fail later.”
↩︎ Where testable transformation code lives“you use a Databricks job to test your shared code and run your notebooks automatically”
↩︎ Integration tests: pipelines and jobs against real tables - 3.
“Databricks follows pytest naming conventions to detect test files, classes and cases.”
↩︎ Where testable transformation code lives“In a pipeline, Declarative Automation Bundles, or Git folder space with an attached cluster, test discovery covers all files in the space's folder.”
↩︎ Where testable transformation code lives - 4.
“Concise syntax for chaining custom transformations.”
↩︎ Composing steps with DataFrame.transform“df.transform(add_n, 1).transform(add_n, n=10).show()”
↩︎ Composing steps with DataFrame.transform“a function that takes and returns a DataFrame.”
↩︎ Key concept“transform(func: Callable[..., "DataFrame"], *args: Any, **kwargs: Any)”
↩︎ Exam trap 3 - 5.https://docs.databricks.com/aws/en/ldp/unit-testingOfficial docs
“from pyspark.testing import assertDataFrameEqual”
↩︎ assertDataFrameEqual and assertSchemaEqual defaults“The framework provides a SparkSession that redirects table operations to a temporary test schema in the pipeline's default catalog”
↩︎ Integration tests: pipelines and jobs against real tables“CAN RUN and CAN MANAGE are not sufficient to run tests.”
↩︎ Integration tests: pipelines and jobs against real tables“Pipeline must run on Databricks Runtime 18.1 or above.”
↩︎ Integration tests: pipelines and jobs against real tables“Test isolation covers table operations that reference a table by name.”
↩︎ Integration tests: pipelines and jobs against real tables“do not read from connectors such as Kafka or Auto Loader. These bypass isolation and act on real production systems.”
↩︎ Exam trap 5 - 6.https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.testing.assertDataFrameEqual.htmlSecondary source
“maxErrors=None, showOnlyDiff=False, includeDiffRows=False”
↩︎ assertDataFrameEqual and assertSchemaEqual defaults“pyspark.testing.assertDataFrameEqual(actual, expected, checkRowOrder=False, rtol=1e-05, atol=1e-08, ignoreNullable=True”
↩︎ Exam trap 1“ignoreNullable=True, ignoreColumnOrder=False, ignoreColumnName=False, ignoreColumnType=False”
↩︎ Exam trap 2“checkRowOrder=False”
↩︎ Prediction“checkRowOrder=False, rtol=1e-05, atol=1e-08, ignoreNullable=True, ignoreColumnOrder=False, ignoreColumnName=False, ignoreColumnType=False”
↩︎ Checkpoint - 7.
“StructField('age', LongType(), True)”
↩︎ assertDataFrameEqual and assertSchemaEqual defaults - 8.
“you might use pytest to test your functions that accept and return PySpark DataFrames in local memory”
↩︎ Spark sessions for unit tests - 9.
“Databricks Connect and PySpark are mutually exclusive.”
↩︎ Spark sessions for unit tests“When running Databricks Connect from the terminal, pytest only works with the DEFAULT configuration profile.”
↩︎ Exam trap 4“When running Databricks Connect from the terminal, pytest only works with the DEFAULT configuration profile.”
↩︎ Checkpoint