Running Databricks Tests: pytest, Notebooks and Pipeline Integration Tests

1.Decide where functions and tests live

A test framework can only discover and run tests that live somewhere it can reach. Databricks describes three ways to arrange functions and their unit tests. Each one trades reuse against the number of files you have to maintain, and the right choice depends on the notebook language.

Databricks also recommends writing and running unit tests from a notebook rather than the web terminal. The web terminal has more limitations, including no Spark support.

Sources1

2.pytest naming, the Tests pane and Databricks Connect

If your tests live in files outside notebooks, pytest's naming conventions decide what runs. When you run pytest, it picks up files named test_*.py or *_test.py in the current directory and its subdirectories. Inside those files it looks for functions whose names start with test_.

The workspace applies the same conventions. When you open a valid Python test file, a Tests icon appears in the left sidebar. It recognises test-prefixed functions, and test-prefixed methods inside Test-prefixed classes that have no __init__ method. The pane lets you run all tests, rerun only the failures, or run a single test, and it marks each test as passed or failed. If a test fails, an inline indicator on the failing line opens the full error message. Discovery scope depends on location. In a pipeline, a Declarative Automation Bundle, or a Git folder with an attached cluster, discovery covers every file in the folder. Everywhere else it covers only the current file.

pytest can also run on your own machine against Databricks compute through Databricks Connect, for Databricks Runtime 13.3 LTS and above. Databricks Connect and PySpark are mutually exclusive in the same environment. When you run from the terminal, pytest only works with the DEFAULT configuration profile, and that profile names the cluster or serverless compute to use. The documented example tests a function that reads samples.nyctaxi.trips, a real workspace table, and asserts that it returns at least one row. That makes it a test of code and data together, not a test of an isolated function.

def get_spark() -> SparkSession:
  spark =  ? .builder.getOrCreate()
  return spark

Sources23

3.Testing inside notebooks

Not every test needs pytest. Many testing libraries run directly in a notebook, including Python's built-in unittest package. Failures appear in the cell's output area.

def reverse(s):
    return s[::-1]

import unittest

class TestHelpers(unittest.TestCase):
    def test_reverse(self):
        self.assertEqual(reverse('abc'), 'cba')

r = unittest.main(argv=[''], verbosity=2, exit=False)
assert r.result.wasSuccessful(), 'Test failed; see logs above'

There are three documented ways to keep test code from getting in the way of normal code. Widgets let one notebook switch between test runs and normal runs: a Mode dropdown decides whether the cell runs asserts or does the real work. The %run magic moves the tests into a separate notebook that pulls in the shared code. Databricks Git folders let tests live in non-notebook source files. To run tests automatically, schedule the notebook as a job with email notifications. For notebooks in a Git folder, you can run the tests on every commit with Databricks GitHub Actions.

Sources4

4.Integration-style tests across pipeline tables

Databricks' notebook unit-testing article explicitly excludes integration testing. For pipelines, the tool that tests how tables work together is the Lakeflow pipeline unit testing framework, which is in Beta. It provides a test SparkSession that sends table operations to a temporary test schema in the pipeline's default catalog. You mock inputs by table name, run part or all of the pipeline graph on the pipeline's compute, and check the output tables with ordinary pytest asserts. One documented use case is testing chains such as bronze, silver and gold, to confirm that data moves correctly through the graph.

# Mock data fixture
def mock_users(session):
    session.sql("""
        CREATE TABLE catalog.schema.wanderbricks_users AS
        SELECT * FROM VALUES
            (1, 'alice@example.com', 'Alice', 'admin'),
            (2, NULL, 'Bob', 'user'),
            (3, 'charlie@example.com', 'Charlie', 'user'),
            (4, NULL, 'Dana', 'admin')
        AS t(user_id, email, name, user_type)
    """)

# Test 1: Row count
def test_users_row_count(test_spark):
    mock_users(test_spark)
    status = test_pipeline.run(test_spark, set(["catalog.schema.users"]))
    assert status.is_success
    result = test_spark.table("catalog.schema.users")
    assert result.count() == 4

test_pipeline is TestPipeline.active(), imported from pyspark.pipelines.testing. Pass it a set of table names to run just those outputs. To test a dependent table such as counts, include the table it reads from as well. Call test_pipeline.run(test_spark) with no set to run the whole pipeline. Besides plain tables, the framework can also validate Auto CDC flows, expectations and append flows.

Isolation applies only to operations that reference a table by name. Reading or writing by path, or reading from connectors such as Kafka or Auto Loader, goes around the isolation and hits production. That includes any upstream flow the selected outputs depend on. Do not run GRANT, REVOKE or ownership statements, and do not create catalogs or schemas, inside tests.

Sources15

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Pipeline unit tests are isolated, so it is safe to run outputs whose upstream flow reads from Auto Loader or a /Volumes path.Why is that wrong?

   Isolation covers only table operations by name. Path-based reads and connectors go around it and act on production systems, including in transitive dependencies.

   Covered in Integration-style tests across pipeline tables

2. 2.CAN MANAGE on a pipeline is enough to run its unit tests.Why is that wrong?

   Only the pipeline Owner can run tests. You also need USE CATALOG and CREATE SCHEMA on the default catalog, so the framework can create its temporary test schema.

   Covered in Integration-style tests across pipeline tables

3. 3.The workspace Tests pane always discovers tests across the whole folder.Why is that wrong?

   Folder-wide discovery happens only in a pipeline, a Declarative Automation Bundle, or a Git folder with an attached cluster. Everywhere else, discovery covers only the current file.

   Covered in pytest naming, the Tests pane and Databricks Connect

Practise it for real

Run a unittest suite inside a Databricks notebook, watch it fail, and then move the code under test into a separate notebook with %run.

1. 1.In a Python notebook, paste the reverse function and the TestHelpers unittest class from the notebook-testing section into one cell, then run it.

   Why: This shows that a built-in test library runs directly in a notebook cell.

   You should see: Verbose unittest output for test_reverse, which passes, and no assertion error.

2. 2.Change the expected value 'cba' to 'abc' and run the cell again.

   Why: You want to see how a failure is reported.

   You should see: The failure appears in the cell's output area, and the final assert raises 'Test failed; see logs above'.

3. 3.Restore 'cba'. In a new cell, add dbutils.widgets.dropdown("Mode", "Test", ["Test", "Normal"]) and an if/else that asserts reverse('abc') == 'cba' in Test mode and prints reverse('desrever') otherwise. Switch the dropdown to Normal.

   Why: Widgets let one notebook separate test runs from normal runs.

   You should see: In Test mode it prints 'Tests passed'. In Normal mode it prints 'reversed'.

4. 4.Move reverse into a notebook named shared-code-notebook in the same folder. Create shared-code-notebook-test whose first cell is %run ./shared-code-notebook and whose second cell holds the unittest code.

   Why: %run keeps the test code in a separate notebook from the code it tests.

   You should see: The test notebook finds reverse through %run, and test_reverse passes.