Bundle Variables and Substitutions for Environment-Specific Configuration

1.Substitutions: values the bundle already knows

When you promote the same codebase from dev to test to prod, most of the configuration stays the same. What changes is where it is deployed, what it is named, and which compute it uses. Declarative Automation Bundles (formerly Databricks Asset Bundles) handle this with two features: substitutions and custom variables. Both let settings be determined at the time a bundle is deployed and run, not hard-coded.

Substitutions read values the bundle already has, such as its name, the target being deployed, or the current user. The most important one for environment-specific configuration is ${bundle.target}. It resolves to the name of the target you deploy to, so one line of YAML gives each environment its own path.

bundle:
  name: hello-bundle

workspace:
  root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/my-envs/${bundle.target}

targets:
  dev:
    default: true

If someone@example.com deploys this, the root path becomes /Workspace/Users/someone@example.com/.bundle/hello-bundle/my-envs/dev. Substitutions can also point at fields of other resources in the same bundle, using forms like ${resources.jobs.<job-name>.id} or ${resources.pipelines.<pipeline-name>.name}.

Sources12

2.Custom variables: values you choose

Substitutions only cover values the bundle already knows. For anything else, such as a cluster ID, a notebook path, or a catalog name that differs between environments, you declare a custom variable. You declare custom variables in the top-level variables mapping. Each variable can have a description and a default.

variables:
  my_cluster_id:
    description: The ID of an existing cluster.
    default: 1234-567890-abcde123
  my_notebook_path:
    description: The path to an existing notebook.
    default: ./hello.py

If a variable has no default, you must supply its value some other way before the bundle commands can use it. The next section covers those ways.

resources:
  jobs:
    hello-job:
      name: hello-job
      tasks:
        - task_key: hello-task
          existing_cluster_id: ${ ? .my_cluster_id}
          notebook_task:
            notebook_path: ${var.my_notebook_path}

Timing matters here. Variables are resolved when you deploy. When you run a job, you run the job that was already deployed, with the values from that deployment. Passing a different variable value at run time does not change it. To vary a single run, use job parameters.

Model answer: bundle variables are interpreted when the bundle is deployed. Running a job runs the previously deployed job with the variable values from that deployment, so a different value passed for the run does not apply. To change something for a single run, the teammate should pass the value as a job parameter instead.

Sources1

3.Setting a variable's value, and which value wins

Besides the default, there are four places to give a variable a value. Together they let the same configuration files serve every environment.

The targets-mapping approach is the core of environment promotion. Each target gives the same variable its own value, for example one cluster ID for dev and another for prod. The resources that use ${var.my_cluster_id} stay exactly the same. The variable is declared once in the top-level variables mapping, as in the previous section, and each target sets a value using the form below.

variables:
      <defined-variable-name>: <non-default-value> # value for this target

The overrides file is a JSON file at .databricks/bundle/<target>/variable-overrides.json, one per target. It holds variable names and values, and it can hold complex values too. The example below shows a complex variable named my_cluster in that file.

{
  "my_cluster": {
    "spark_version": "13.2.x-scala2.11",
    "node_type_id": "Standard_DS3_v2",
    "num_workers": 2
  }
}

When more than one source sets a value, the CLI checks them in a fixed order and stops at the first value it finds.

Sources13

4.Complex variables, lookups, and checking resolved values

Variables are strings by default. To pass a whole structure, such as a cluster definition that should be larger in prod, set type: complex and give a structured default. You can then use it in a job like new_cluster: ${var.my_cluster}. You can also define complex values in variable-overrides.json.

A lookup variable solves a different problem. Object IDs often differ between workspaces, but names can stay the same. A lookup takes the name of an object and resolves it to that object's ID. Lookups are available for these object types: alert, cluster_policy, cluster, dashboard, instance_pool, job, metastore, notification_destination, pipeline, query, service_principal, and warehouse.

variables:
  my_cluster_id:
    description: An existing cluster
    lookup:
      cluster: '12.2 shared'

The lookup fails with an error if no object has that name, or if more than one does. Before deploying anywhere, run databricks bundle validate to make sure your substitutions and variables parse. Add --output json to see the values that will actually be used. The output shows each variable's default next to its resolved value.

Sources1

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Passing --var to databricks bundle run changes the variable values that the already-deployed job uses.Why is that wrong?

   Variables are resolved at deploy time. A run uses the job as it was deployed, so to vary a single run, pass job parameters.

   Covered in Custom variables: values you choose

2. 2.A value set in a target's variables mapping overrides a BUNDLE_VAR_ environment variable, because the target is more specific.Why is that wrong?

   The CLI checks --var first, then BUNDLE_VAR_ environment variables, then variable-overrides.json, then the targets mapping, and finally the default.

   Covered in Setting a variable's value, and which value wins