Python code in bundles: wheel artifacts, library dependencies, and Python-defined resources

1.Building your src/ package as a wheel artifact

In a scalable bundle, shared logic belongs in a Python package under src/ that is built into a wheel, not copied between notebooks. The default-python template already does this: it builds the wheel with uv from a pyproject.toml. The build is set in the top-level artifacts mapping in databricks.yml.

# databricks.yml
...
artifacts:
  python_artifact:
    type: whl
    build: uv build --wheel
...

artifacts:
  python_artifact:
    type:  ? 
    build: uv build --wheel

uv is not the only option. The documentation also shows Poetry (build: poetry build) and setuptools builds that keep the package under src/my_package with __init__.py, main.py and my_module.py. The task then runs the package through a python_wheel_task and lists the built file under libraries as ./dist/*.whl.

Deploying the bundle builds the wheel. The built file goes into the bundle's workspace folder under .bundle/<project-name>/dev/artifacts/.internal/. If the wheel was already built somewhere else, such as an earlier CI step, you leave out the artifacts mapping.

Sources1

2.Declaring library dependencies on tasks

Jobs and pipelines also depend on outside libraries, and these are declared in the bundle configuration too. Python dependencies on a job task go in its libraries list as wheel files, PyPI packages or a requirements file. JARs and Maven or CRAN packages are also supported.

resources:
  jobs:
    my_job:
      # ...
      tasks:
        - task_key: my_task
          # ...
          libraries:
            - whl: ./my-wheel-0.1.0.whl
            - whl: /Workspace/Shared/Libraries/my-wheel-0.0.1-py3-none-any.whl
            - whl: /Volumes/main/default/my-volume/my-wheel-0.1.0.whl

Where the files are stored matters. Storing libraries in the DBFS root is deprecated and turned off by default from Databricks Runtime 15.1, because any workspace user can change files there. Databricks recommends workspace files, Unity Catalog volumes or package repositories instead.

Sources2

3.pyproject.toml with uv, and the pipeline exception

The recommended way to manage Python dependencies is uv, with dependencies listed in the bundle's pyproject.toml. The template's pyproject keeps runtime dependencies apart from a dev dependency group used for local work (pytest, ruff, databricks-dlt, databricks-connect, ipykernel). The deployed code then needs only the runtime set.

To give a pipeline every dependency in pyproject.toml, add an editable environment to the pipeline resource that points to the folder where pyproject.toml is deployed. The example also shows two other structural habits: catalog and schema come from bundle variables, and the source is picked up with a glob over the transformations/ folder.

resources:
  pipelines:
    test_uv_etl:
      name: test_uv_etl
      catalog: ${var.catalog}
      schema: ${var.schema}
      serverless: true
      root_path: '../src/test_uv_etl'

      libraries:
        - glob:
            include: ../src/test_uv_etl/transformations/**

      environment:
        dependencies:
          - --editable ${workspace.file_path}

Sources2

4.Defining resources in Python

Resource definitions do not have to be YAML. With Python support for bundles you can define resources in Python code alongside YAML resources. You can also generate resources from metadata during deployment, and change YAML- or Python-defined resources while deploying. It needs Databricks CLI 0.275.0 or above. The databricks bundle init pydabs template sets it up.

from databricks.bundles.jobs import Job, Task, NotebookTask


my_pydabs_project_job = Job(
    name="my_pydabs_project_job",
    tasks=[
        Task(
            task_key="notebook_task",
            notebook_task=NotebookTask(
                notebook_path="src/notebook.ipynb",
            ),
        ),
    ],
)

Job.from_dict accepts the same structure as a Python dictionary. databricks.yml connects these files through a python section. That section names the virtual environment, which must contain the databricks-bundles package, and the function that loads the resources.

python:
  # Activate the virtual environment before loading resources defined in
  # Python. If disabled, it defaults to using the Python interpreter
  # available in the current shell.
  venv_path: .venv
  # Functions called to load resources defined in Python.
  # See resources/__init__.py
  resources:
    - 'resources:load_resources'

By default, load_resources in resources/__init__.py loads every Python file in the resources package. The CLI calls it during deployment, and it is not used after that. Because it is ordinary Python, it can build resources in a loop. This is a scalable pattern when many jobs differ only by a parameter.

def load_resources(bundle: Bundle) -> Resources:
    resources = load_resources_from_current_package_module()


    for country in ["US", "NL"]:
        resources.add_resource(f"my_job_{country}", create_job(country))


    return resources

Sources3

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.If a bundle has no artifacts mapping, no wheel is deployed for its python_wheel_task.Why is that wrong?

   Without artifacts, the CLI treats the wheel as already built and deploys the whl files listed under libraries. Leaving out the mapping is the documented way to deploy a wheel built elsewhere.

   Covered in Building your src/ package as a wheel artifact

2. 2.Adding a package to pyproject.toml is enough for a bundle-deployed pipeline to use it during development.Why is that wrong?

   Pipeline dependencies are cached during development. Declare them in the pipeline's environment section, for example as an editable install of the deployed project.

   Covered in pyproject.toml with uv, and the pipeline exception

3. 3.Uploading wheels to the DBFS root is still the standard place for bundle libraries.Why is that wrong?

   Libraries in the DBFS root are deprecated and turned off by default on newer runtimes. Use workspace files, Unity Catalog volumes or package repositories.

   Covered in Declaring library dependencies on tasks

Practise it for real

Create a default-python bundle that builds a wheel from src/, deploy it to the dev target, and run its job

1. 1.Run databricks bundle init, keep the default-python template, answer no to the notebook job and ETL pipeline, yes to the sample Python package and serverless, and enter an existing Unity Catalog catalog

   Why: The template produces the standard layout: databricks.yml, resources/sample_job.job.yml, src/ and tests/

   You should see: A new project folder (my_project by default) with those files

2. 2.From the bundle root, run databricks bundle validate

   Why: Validation catches configuration errors before anything reaches the workspace

   You should see: A summary of the bundle configuration and no errors

3. 3.Run databricks bundle deploy --target dev

   Why: Deploying builds the wheel from src/ and creates the job in development mode

   You should see: The .whl under .bundle/<project-name>/dev/artifacts/.internal and a job named [dev <your-username>] sample_job with one python_wheel_task

4. 4.Run databricks bundle run --target dev sample_job and open the printed Run URL

   Why: This confirms the deployed wheel's entry point runs as a job task

   You should see: The task finishes with a green title bar