Declarative Automation Bundles: databricks.yml, Targets, Overrides and Variables

1.What a bundle packages

Declarative Automation Bundles were formerly called Databricks Asset Bundles, and you will still see both names. They bring software engineering practice to Databricks projects: source control, code review, testing and CI/CD. Without bundles, you create a job in the UI and its definition stays in the workspace. With a bundle, the job definition is a YAML file in the same repository as the notebooks it runs. It is versioned and reviewed with that code, and deployed with it.

According to the documentation, a bundle contains four kinds of things: the cloud infrastructure and workspace configuration it needs; source files such as notebooks and Python files that hold the business logic; definitions and settings for Databricks resources; and unit and integration tests. The resource types listed include Lakeflow Jobs, Lakeflow pipelines, dashboards, Model Serving endpoints, MLflow experiments and MLflow registered models. Databricks describes bundles as an infrastructure-as-code (IaC) approach. They fit best on complex projects with several contributors, where automation and CI/CD are required.

Bundles are a feature of the Databricks CLI. You write the YAML, then use the CLI to validate it, deploy it to a remote workspace and run the deployed workflows. You can also work with bundles directly in the workspace, which does not require installing the CLI.

Sources1

2.databricks.yml: the one root configuration file

Every bundle has exactly one file named databricks.yml at the root of the project folder. That file defines the bundle. It can pull in other YAML files, such as one file per job or pipeline, through the include mapping. The smallest valid configuration has a bundle name, which is required, and one target.

bundle:
  name: my_bundle

targets:
  dev:
    default: true

Lakeflow Jobs go under resources.jobs and Lakeflow pipelines go under resources.pipelines. Each one is keyed by a programmatic identifier that you choose. Other workspace assets have their own keys under resources, including dashboards, schemas, volumes, experiments and model_serving_endpoints. The other top-level mappings set defaults, and each target can override them.

Sources23

3.Targets: one definition, environment-specific overrides

The targets mapping is how one bundle serves dev, test and prod. Each target is a named environment. Settings at the top level are the defaults. Settings inside a target are applied on top of those defaults when you deploy to that target. In the example below, prod points at a different workspace host and swaps the cluster ID. It does not repeat the notebook_task, because anything a target leaves out falls back to the top-level definition.

targets:
  dev:
    default: true
  prod:
    workspace:
      host: https://<production-workspace-url>
    resources:
      jobs:
        hello-job:
          name: hello-job
          tasks:
            - task_key: hello-task
              existing_cluster_id: 2345-678901-fabcd456

Databricks recommends setting a host for each target rather than relying on the default mapping alone, because it makes the configuration more portable. When a host is set, the CLI looks for a profile with that host in your .databrickscfg file and uses it to authenticate.

resources:
  jobs:
    my-job:
      name: my-job
      job_clusters:
        - job_cluster_key: my-cluster
          new_cluster:
            spark_version: 13.3.x-scala2.12

targets:
  development:
    resources:
      jobs:
        my-job:
          name: my-job
          job_clusters:
            - job_cluster_key: my-cluster
              new_cluster:
                node_type_id: i3.xlarge
                num_workers: 1

The rule is to merge first, then let the target win on conflicts. If spark_version and num_workers were set at both levels, the target's values would replace the top-level ones, while node_type_id from the top level would remain. Each kind of setting has its own join key. Job clusters join on job_cluster_key, job tasks on task_key, and Lakeflow pipeline clusters on label (default or maintenance). Artifacts follow the same pattern: an artifact setting defined in a target takes precedence over the top-level one.

Sources42

4.Custom variables for per-environment values

Overriding whole resource blocks works, but often only one value changes between environments, such as a cluster ID or a notebook path. Custom variables handle this. You declare them in the variables mapping or in a variable-overrides.json file, give each a description and an optional default, and reference them anywhere in the configuration.

variables:
  my_cluster_id:
    description: The ID of an existing cluster.
    default: 1234-567890-abcde123
  my_notebook_path:
    description: The path to an existing notebook.
    default: ./hello.py

To use a variable, write ${var.my_cluster_id}. To give a target its own value, set it under that target's variables mapping. The configuration specification shows this as <defined-variable-name>: <non-default-value> # value for this target. Bundles also provide built-in substitutions, such as ${bundle.target}, ${bundle.name}, ${workspace.host} and ${workspace.current_user.short_name}, so names and paths can include the environment or the deploying user.

Sources52

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Defining new_cluster under a target replaces the entire top-level new_cluster block.Why is that wrong?

   Settings with the same join key are merged. The target value wins only for a field defined at both levels, and fields defined only at the top level are kept.

   Covered in Targets: one definition, environment-specific overrides

2. 2.Each subfolder of a bundle can have its own databricks.yml for its jobs or pipelines.Why is that wrong?

   A bundle has exactly one databricks.yml, at the project root. Other YAML files are added through the include mapping.

   Covered in databricks.yml: the one root configuration file