What you will be able to do
- Define dev, test and prod targets that reuse one set of top-level resource definitions and override only what differs
- Use built-in substitutions such as ${bundle.target} and custom ${var.name} variables to make one job definition environment-aware
- Predict which value a bundle variable resolves to when it is set in several places at once
- Declare complex variables with type: complex and lookup variables that resolve an object's name to its ID
- Explain how target-level resource settings merge with top-level ones, and how deployment modes and presets change a target's behaviour
- Configure run_as so that dev resources run as the developer and prod resources run as a service principal
Key concept
Target as an overlay — A bundle defines its resources once at the top level. Each target (dev, test, prod) adds or replaces only the settings that differ for that environment, and variables and substitutions fill in the rest when the bundle is deployed. Any setting a target leaves out falls back to the top-level definition.
1.One codebase, several targets
Promoting code from dev to test to prod only works if all three environments run the same definitions. If you copy a job's YAML once per environment, the copies start to differ the first time someone fixes one and forgets the others. Declarative Automation Bundles (formerly Databricks Asset Bundles) avoid this with targets. The top-level resources mapping in databricks.yml defines each job and pipeline once, and the top-level targets mapping lists the places those definitions get deployed.
A target is a named deployment destination. Each one can carry its own workspace, resources, variables, mode, presets, permissions and run_as settings. Exactly one target may be marked default: true; that one is used when a command names no target, and every other target has to be selected explicitly, for example with -t prod (see selecting a target with the CLI).
targets:
dev:
default: true
prod:
workspace:
host: https://<production-workspace-url>
resources:
jobs:
hello-job:
name: hello-job
tasks:
- task_key: hello-task
existing_cluster_id: 2345-678901-fabcd456The prod target above never mentions notebook_task, but the prod job still runs ./hello.py. Settings a target leaves out fall back to the top-level definition, so a target only lists what is different about its environment. The host mapping is also worth noting. When you set it, the CLI looks for a matching profile in .databrickscfg. Databricks recommends host over relying on the default profile because it makes the configuration portable.
Checkpoint 1 of 11· Fill the gap
In this databricks.yml, databricks bundle deploy with no -t flag should deploy to dev, while prod is deployed only when selected with -t prod. Which key completes the dev target?
targets:
dev:
? : true
prod:
workspace:
host: https://<production-workspace-url>
resources:
jobs:
hello-job:
name: hello-job
tasks:
- task_key: hello-task
existing_cluster_id: 2345-678901-fabcd456Only one target can be marked default: true, and the CLI uses that target when a command names no target. Any other target, like prod here, has to be chosen with -t. mode takes a string such as development or production, not true, and it doesn't control which target the CLI selects.
Sources1
2.Built-in substitutions resolve at deploy time
Substitutions are ${...} references to values the bundle already knows: its name, the target being deployed, the workspace host, the deploying user, and fields of its own resources. They are filled in when the bundle is deployed and run. That lets one line of YAML produce a different value in every target without you writing that value anywhere.
bundle:
name: hello-bundle
workspace:
root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/my-envs/${bundle.target}
targets:
dev:
default: trueIf someone@example.com deploys this to dev, the files land in /Workspace/Users/someone@example.com/.bundle/hello-bundle/my-envs/dev. If another engineer deploys it, or if it is deployed to a different target, the path changes and nobody edits the file.
Resources can also refer to each other's fields. A pipeline whose schema is set to pipeline_bundle_${bundle.target} can be referenced from another resource as ${resources.pipelines.my_pipeline.schema}, which resolves to pipeline_bundle_dev when you deploy to dev. The substitutions you will see most often are:
| Substitution | Resolves to |
|---|---|
| ${bundle.name} | The bundle's name |
| ${bundle.target} | The target being deployed. Use this instead of ${bundle.environment} |
| ${workspace.host} | The workspace URL |
| ${workspace.current_user.userName} / ${workspace.current_user.short_name} | The identity running the bundle command |
| ${workspace.root_path} / ${workspace.file_path} | Where the bundle is deployed in the workspace |
| ${resources.jobs.<job-name>.id} | The ID of a job the bundle deployed |
Checkpoint 2 of 11· Check yourself
A teammate, other@example.com, adds a prod target to the hello-bundle shown above and runs databricks bundle deploy -t prod. The databricks.yml is unchanged otherwise. Where do the bundle's files land in the workspace?
Built-in substitutions resolve at deploy time, so ${workspace.current_user.userName} becomes the person deploying and ${bundle.target} becomes the target chosen with -t. default: true applies only when no target is named, and ${bundle.target} is a built-in value, so no custom variable is needed.
“subsitutions can be used to refer to the values of the bundle name, bundle target, and workspace userName fields to construct the workspace root_path”Source: docs.databricks.com
Sources2
3.Custom variables with a value per target
Substitutions only cover values the bundle already knows. Some values exist only in your organisation, such as the Unity Catalog catalog a job writes to or the cluster it uses, and these need custom variables. You declare a variable in the top-level variables mapping, optionally with a description and a default, and reference it anywhere as ${var.<variable_name>}.
variables:
my_cluster_id:
description: The ID of an existing cluster.
default: 1234-567890-abcde123
my_notebook_path:
description: The path to an existing notebook.
default: ./hello.pyresources:
jobs:
hello-job:
name: hello-job
tasks:
- task_key: hello-task
existing_cluster_id: ${var.my_cluster_id}
notebook_task:
notebook_path: ${var.my_notebook_path}To make a value environment-specific, give each target its own value under targets.<name>.variables, for example my_cluster_id set to one cluster under dev and another under prod. A job that must write to a different catalog per environment follows the same pattern. Declare a catalog variable with a default, use ${var.catalog} in the task configuration, and override the value per target. The job is still defined only once. Duplicating it into dev_job and prod_job brings back the drift that targets exist to prevent. A workspace-wide default can't help either, because it is the same for every target that deploys to that workspace.
If a variable has no default, a value has to come from somewhere else: a command-line flag, an environment variable, a target mapping, or the .databricks/bundle/<target>/variable-overrides.json file. Otherwise the variable is unresolved.
Checkpoint 3 of 11· Check yourself
A job writes to a Unity Catalog catalog that must be dev_catalog for dev, test_catalog for test and prod_catalog for prod. Dev and test deploy to the same workspace. Which approach keeps a single job definition and still gives each target its own catalog?
A custom variable is declared once and referenced as ${var.catalog}, and each target sets its own value in its own variables mapping, so the job is defined only once. Copying the job brings back drift between the copies. A workspace default can't tell dev and test apart because they share a workspace. A widget value is read at run time, outside bundle configuration, so targets can't set it.
“Provide the variable's value within your bundle configuration files using the variables mapping within the targets mapping”Source: docs.databricks.com
Checkpoint 4 of 11· Exam question
A data engineering team maintains a single job definition in an Automation Bundle that must write to a different Unity Catalog catalog depending on whether it is deployed to the dev or prod target, without duplicating the job's YAML. What should they do?
Correct answer: A — Declare a catalog variable with a default value, reference ${var.catalog} in the job's task config, and override it per target in databricks.yml.
- A. Declaring a
catalogvariable with a default and referencing${var.catalog}in the task config lets the single job definition stay unchanged while each target block supplies its own catalog value. This is the standard bundle pattern for environment-specific configuration without duplicating resources. - B. Duplicating the job into two hardcoded resources defeats the purpose of a shared bundle codebase and forces the team to keep two YAML definitions in sync manually. Any future change to the job logic now has to be made twice.
- C. Reading the catalog name from a notebook widget at runtime bypasses bundle configuration entirely and is not how Automation Bundles pass environment-specific values into job tasks. It also makes the value invisible to
bundle validate. - D. A workspace-wide Unity Catalog default is a governance setting, not a bundle mechanism, and it cannot differentiate between the
devandprodtargets of the same bundle deployment. It would apply the same catalog everywhere regardless of target.
Sources2
4.Which value wins when a variable is set twice
A variable can be given a value in five places, and in real projects more than one is often set. A CI runner might export an environment variable while databricks.yml also has a per-target value. The CLI checks the five places in a fixed order and stops at the first value it finds.
| Order | Source of the value | Form |
|---|---|---|
| 1 | --var option on the bundle command | --var="my_cluster_id=1234-567890-abcde123" |
| 2 | Environment variable | BUNDLE_VAR_my_cluster_id=1234-567890-abcde123 |
| 3 | variable-overrides.json, if it exists | { "my_cluster_id": "1234-567890-abcde123" } in .databricks/bundle/<target>/ |
| 4 | variables mapping inside the target | my_cluster_id: 2345-678901-bcdef234 |
| 5 | default in the top-level variables mapping | default: 1234-567890-abcde123 |
Everything that comes from outside the YAML beats everything inside it. That lets a pipeline or an engineer override a committed value without editing the file. It also means a forgotten BUNDLE_VAR_ export on a build agent quietly overrides what the prod target says. Pick one way of supplying values and use it for both the deploy and the run. If you mix them, the run may be based on a deployment made with different values.
You don't have to work the order out by hand. databricks bundle validate --output json prints the resolved configuration, and each variable shows its default next to the value that will actually be used.
Checkpoint 5 of 11· Put it in order
A prod deployment could get my_cluster_id from any of these five places. Put them in the order the Databricks CLI checks them, first to last. The CLI stops at the first one that has a value.
- 1.The default under the top-level variables.my_cluster_id mapping
- 2.A BUNDLE_VAR_my_cluster_id environment variable exported on the runner
- 3.--var="my_cluster_id=..." passed on the bundle command
- 4.my_cluster_id in .databricks/bundle/prod/variable-overrides.json
- 5.my_cluster_id under targets.prod.variables in databricks.yml
Values from outside the YAML come first: the --var flag, then BUNDLE_VAR_ environment variables, then variable-overrides.json. After those comes the target's variables mapping, and the top-level default is used only when nothing else sets a value. So a value written for the prod target can still lose to an exported environment variable.
“The Databricks CLI looks for values for variables in the following order, stopping when it finds a value for a variable:”Source: docs.databricks.com
Sources2
5.Complex variables and lookups
Custom variables are strings unless you say otherwise. Some values are structured, such as a whole cluster specification that should differ between dev and prod. For those, set type: complex and give the variable an object as its default. complex is the only valid value for type. There is no object type, and leaving type out means the variable is a string. Validation also fails if a variable is declared type: complex but its default is a single value.
variables:
my_cluster:
description: 'My cluster definition'
type: complex
default:
spark_version: '13.2.x-scala2.11'
node_type_id: 'Standard_DS3_v2'
num_workers: 2
spark_conf:
spark.speculation: true
spark.databricks.delta.retentionDurationCheck.enabled: false
resources:
jobs:
my_job:
job_clusters:
- job_cluster_key: my_cluster_key
new_cluster: ${var.my_cluster}A complex value can also be supplied per target in .databricks/bundle/<target>/variable-overrides.json as a JSON object with the same keys.
Lookup variables handle a different problem: object IDs that differ from one workspace to the next. Instead of hardcoding a cluster ID for each environment, you name the object and the CLI finds its ID. Lookups work for alerts, cluster policies, clusters, dashboards, instance pools, jobs, metastores, notification destinations, pipelines, queries, service principals and warehouses.
variables:
my_cluster_id:
description: An existing cluster
lookup:
cluster: '12.2 shared'The name has to match exactly one object. If no object has that name, or if two do, the deployment fails with an error. A failed deployment is better than one that quietly uses the wrong cluster.
Checkpoint 6 of 11· Fill the gap
Dev and prod each need a different whole cluster spec, so my_cluster has to hold a structured default instead of a string. Bundle validation should accept this definition. What goes in the blank after type:?
variables:
my_cluster:
description: 'My cluster definition'
type: ?
default:
spark_version: '13.2.x-scala2.11'
node_type_id: 'Standard_DS3_v2'
num_workers: 2
spark_conf:
spark.speculation: true
spark.databricks.delta.retentionDurationCheck.enabled: false
resources:
jobs:
my_job:
job_clusters:
- job_cluster_key: my_cluster_key
new_cluster: ${var.my_cluster}complex is the only valid value for a variable's type, and you need it to give the variable an object as its default. There is no object, map or dictionary type. If you leave type out, the variable is treated as a string and is not inferred from the default's structure.
Checkpoint 7 of 11· Exam question
How does an engineer declare a bundle variable in databricks.yml whose default value is a structured object, such as a cluster spec with a Spark version and worker count, rather than a single string?
Correct answer: A — Set type: complex on the variable and give it an object as its default value, such as a mapping of Spark version and worker count.
- A. Setting
type: complextells the bundle schema to accept a structured object as the variable's value, and thedefaultkey is where that object, such as a Spark version and worker count mapping, is supplied. This is the documented way to define complex bundle variables. - B.
type: objectis not a recognized bundle variable type, and splitting nested keys into separate top-level variables with a naming convention does not reconstruct a single structured value at reference time. The bundle schema only supportscomplexfor this purpose. - C. Omitting
typecauses the bundle parser to treat the variable as a plain string rather than inferring a nested structure from a YAML anchor. Thetype: complexdeclaration is required for the parser to accept an object as the default. - D. A
--varflag only accepts simple string values on the command line and cannot express a nested object, so a complex variable's structured default must still be defined insidedatabricks.ymlitself. Thedefaultkey is where that structure belongs, not a CLI flag.
Sources2
6.How target resource settings merge with top-level ones
When a target redefines a resource, its settings are merged with the top-level definition. The target does not replace the whole resource. Items in a list are matched by key: job clusters by job_cluster_key, job tasks by task_key, pipeline clusters by label, and artifacts by their identifier. The key has to be the same in both places or nothing gets merged. For each matched item, settings that exist on only one side are combined, and if a setting exists on both sides, the target's value wins.
resources:
jobs:
my-job:
name: my-job
job_clusters:
- job_cluster_key: my-cluster
new_cluster:
spark_version: 13.3.x-scala2.12
targets:
development:
resources:
jobs:
my-job:
name: my-job
job_clusters:
- job_cluster_key: my-cluster
new_cluster:
node_type_id: i3.xlarge
num_workers: 1Validated for development, my-cluster has all three settings. Pipelines behave the same way. Suppose the top level gives the default cluster node_type_id: i3.xlarge and num_workers: 1, and the target sets num_workers: 2. The result keeps the node type and uses two workers. In practice the top level holds the shared definition, and each target writes down only the few settings that differ, usually sizing, cluster IDs and hosts.
Checkpoint 8 of 11· Fill the gap
For the development target, validation should show one cluster, my-cluster, with spark_version taken from the top level and node_type_id and num_workers taken from the target. Which value completes the target's cluster entry?
resources:
jobs:
my-job:
name: my-job
job_clusters:
- job_cluster_key: my-cluster
new_cluster:
spark_version: 13.3.x-scala2.12
targets:
development:
resources:
jobs:
my-job:
name: my-job
job_clusters:
- job_cluster_key: ?
new_cluster:
node_type_id: i3.xlarge
num_workers: 1Job clusters are matched by job_cluster_key. The target's entry merges with the top-level cluster only if it uses the same key, my-cluster. A different key such as the job name, the target name or a per-target suffix doesn't match, so nothing is merged and the cluster's settings aren't combined.
Source: docs.databricks.comSources3
7.Deployment modes and presets per target
Setting mode on a target applies a group of common behaviours in one step. Modes are optional. mode: development is meant for a developer's own iteration loop. mode: production adds safety checks before a release. See deploying bundles for the deploy step itself.
| Behaviour | mode: development | mode: production |
|---|---|---|
| Resource names | Prefixed with [dev ${workspace.current_user.short_name}] and tagged dev | Unchanged |
| Lakeflow pipelines | Marked development: true | Validated as development: false |
| Schedules and triggers | Paused | Unchanged |
| Cluster override with --cluster-id | Allowed | Not allowed |
| Git branch | Not checked | Must match the target's git branch, if one is set (--force skips the check) |
| Deployment lock | Disabled | Unchanged |
The development prefix is itself a substitution, so two engineers deploying the same bundle to dev get separately named jobs and don't overwrite each other's. When a mode's defaults don't suit you, set presets on the target, for example name_prefix, trigger_pause_status or jobs_max_concurrent_runs. Presets override the mode, and settings on an individual resource override the presets. A job that explicitly sets max_concurrent_runs: 10 keeps 10 even when jobs_max_concurrent_runs is set to 20.
Checkpoint 9 of 11· Check yourself
The dev target sets mode: development and also has presets with trigger_pause_status: PAUSED. One job in the bundle sets its own schedule pause_status to UNPAUSED. After databricks bundle deploy -t dev, what is that job's schedule state?
There are three layers. The mode sets defaults, presets override the mode, and a setting on an individual resource overrides the presets. The job's explicit UNPAUSED is the most specific, so it wins and the schedule stays unpaused. The bundle doesn't report this as a conflict.
“if both mode and presets are set, presets override the default mode behavior, and settings of individual resources override the presets.”Source: docs.databricks.com
Sources4
8.Run identity per target with run_as
run_as sets the identity a deployed job or pipeline runs as, which can differ from the identity that deployed it. You can set it as a top-level mapping or inside a target. It takes a user_name or a service_principal_name, and a service principal is always given by its application ID. Non-admins can only set it to their own email.
The usual pattern is for development to run as the individual developer, and for staging and production to run as their own service principals. A top-level run_as gives the default, and each target overrides it:
run_as:
service_principal_name: '5cf3z04b-a73c-4x46-9f3d-52da7999069e'
targets:
# Development deployment settings, set as the default
development:
mode: development
default: true
workspace:
host: https://my-host.cloud.databricks.com
run_as:
user_name: someone@example.com production:
mode: production
workspace:
host: https://my-host.cloud.databricks.com
root_path: /Shared/production-workspace/.bundle/${bundle.name}/${bundle.target}
run_as:
service_principal_name: '68ed9cd5-8923-4851-x0c1-c7536c67ff99'Running prod as a service principal separates permission to run the workflow from the person who deployed it. Either the same service principal deployed the workflow, or the deployer has CAN_USE on that service principal. The principal can also be given fewer permissions than the deploying identity. Two shortcuts don't achieve this. Granting the service principal CAN_MANAGE on a folder after deployment leaves the run identity unchanged. Putting the ID in a custom variable doesn't work either, because only the run_as mapping sets who a resource runs as.
A few limits apply. If the deploying identity differs from the run_as identity, only jobs and pipelines are supported. Top-level and target run_as mappings cannot name a group, although an individual job or pipeline's own run_as can use group_name. A bundle that contains model serving endpoints and also sets run_as fails with an error.
Checkpoint 10 of 11· Fill the gap
Production jobs must run as their own service principal, given by its application ID, while development runs as the individual developer. Which key completes the production target's run_as mapping?
production:
mode: production
workspace:
host: https://my-host.cloud.databricks.com
root_path: /Shared/production-workspace/.bundle/${bundle.name}/${bundle.target}
run_as:
? : '68ed9cd5-8923-4851-x0c1-c7536c67ff99'run_as takes either a user_name or a service_principal_name, and you give a service principal as its application ID. Target-level and top-level run_as mappings can't name a group. There is no service_principal_app_id key, because the application ID goes under service_principal_name.
Source: docs.databricks.comCheckpoint 11 of 11· Exam question
A team wants every resource deployed by their bundle's prod target to run under a service principal instead of the deploying engineer's own identity. How should they configure this in databricks.yml?
Correct answer: A — Add a run_as block under the prod target that specifies the service principal's application ID, so deployed resources run under that identity.
- A. A
run_asblock under theprodtarget is the supported way to make deployed jobs and pipelines execute as a service principal rather than the deploying user. It applies to the resources in that target without touching the job definitions themselves. - B. Granting workspace folder permissions after the fact controls who can view or edit the deployed files, but it does not change which identity the jobs actually run as during execution. The
run_asfield is still required to change the execution identity. - C. The
modefield controls deployment behaviors like resource-name prefixing and schedule pausing, not the execution identity of deployed resources. Setting it toproductionhas no effect on who the jobs run as. - D.
run_asis a target-level (or resource-level) field, not something resolved through a${var.*}reference inside job task definitions, so referencing it that way would not be recognized by the bundle schema. The identity must be set through therun_asblock itself.
Sources
(5)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“These are the targets to use for deployments and workflow runs.”
↩︎ One codebase, several targets“as it falls back to use the notebook_task mapping within the top-level resources mapping”
↩︎ One codebase, several targets“Databricks recommends that you use the host mapping instead of the default mapping wherever possible”
↩︎ One codebase, several targets“as it falls back to use the notebook_task mapping within the top-level resources mapping”
↩︎ Key concept - 2.
“settings can be determined at the time a bundle is deployed and run”
↩︎ Built-in substitutions resolve at deploy time“${resources.pipelines.my_pipeline.schema} resolves to pipeline_bundle_dev when you deploy to the dev target”
↩︎ Built-in substitutions resolve at deploy time“To reference a custom variable within your bundle configuration, use the variable substitution ${var.<variable_name>}.”
↩︎ Custom variables with a value per target“using the variables mapping within the targets mapping”
↩︎ Custom variables with a value per target“Instead, pass values to a job run using job parameters.”
↩︎ Custom variables with a value per target“The Databricks CLI looks for values for variables in the following order, stopping when it finds a value for a variable:”
↩︎ Which value wins when a variable is set twice“use the same approach during both the deployment and run stages”
↩︎ Which value wins when a variable is set twice“To view values that will be used when you deploy a bundle, use the --output json option:”
↩︎ Which value wins when a variable is set twice“The only valid value for the type setting is complex.”
↩︎ Complex variables and lookups“bundle validation fails if type is set to complex and the default defined for the variable is a single value.”
↩︎ Complex variables and lookups“An error occurs if an object with the specified name does not exist”
↩︎ Complex variables and lookups“The Databricks CLI looks for values for variables in the following order, stopping when it finds a value for a variable:”
↩︎ Exam trap 2“The only valid value for the type setting is complex.”
↩︎ Exam trap 3“Bundle variables are deployment-time variables. They are interpreted when you deploy the bundle.”
↩︎ Exam trap 4“subsitutions can be used to refer to the values of the bundle name, bundle target, and workspace userName fields to construct the workspace root_path”
↩︎ Checkpoint“Provide the variable's value within your bundle configuration files using the variables mapping within the targets mapping”
↩︎ Checkpoint“Within any environment variables set that begin with BUNDLE_VAR_.”
↩︎ Prediction - 3.
“then the setting in the targets mapping takes precedence over the setting in the top-level resources mapping.”
↩︎ How target resource settings merge with top-level ones“the task mappings' task_key must be set to the same value”
↩︎ How target resource settings merge with top-level ones - 4.
“Deployment modes are a convenience for applying a group of commonly used settings at once.”
↩︎ Deployment modes and presets per target“presets override the default mode behavior, and settings of individual resources override the presets.”
↩︎ Deployment modes and presets per target“You can enforce this by setting run_as to a service principal.”
↩︎ Run identity per target with run_as“You can enforce this by setting run_as to a service principal.”
↩︎ Exam trap 1“if both mode and presets are set, presets override the default mode behavior, and settings of individual resources override the presets.”
↩︎ Checkpoint - 5.
“Non-admins can only set this field to their own email.”
↩︎ Run identity per target with run_as“Decouples the permission to run the production workflow from the identity that created or deployed the bundle.”
↩︎ Run identity per target with run_as“The top-level and target run_as mappings do not support groups.”
↩︎ Run identity per target with run_as