What you will be able to do
- Wire tasks together with Depends on and predict which tasks run in parallel
- Choose the right Run if condition and predict when a task ends as Upstream failed or Excluded
- Trace how excluded, canceled and disabled upstream tasks affect downstream tasks
- Place If/else, For each and Run Job tasks in a DAG and know the pipeline-in-loop concurrency cap
1.Depends on: drawing the DAG
Dependencies are what turn a set of tasks into an execution order. Each task's Depends on field lists its upstream tasks, and the job DAG draws every dependency as a line between two tasks. The field only appears once the job has more than one task. If a task is selected in the DAG when you create a new task, the new task depends on the selected one by default.
Databricks runs upstream tasks before downstream tasks and runs as many tasks in parallel as it can. Take the four-task example from the docs. Task 1 is the root. Tasks 2 and 3 both depend on Task 1, so they run side by side once it finishes. Task 4 depends on both of them. Tasks that have no dependency on each other don't wait for each other.
Checkpoint 1 of 7· Exam question
A job has a `load_customers` task and a `load_orders` task that both feed into a `reconcile` task, which should proceed only if both upstream loads completed without error. Which `Run if dependencies` setting on `reconcile` matches this requirement?
Correct answer: A — All succeeded, so `reconcile` runs only when every task it depends on completes successfully
- A. This is the default run-if condition and the correct choice here, since it requires every dependency, both the customer load and the order load, to succeed before the downstream task is allowed to start.
- B. This condition only requires one dependency to succeed, so `reconcile` could start even if one of the two loads failed, which does not match the requirement that both loads must succeed.
- C. This condition tolerates a dependency that was skipped or excluded as long as nothing explicitly failed, which is looser than requiring both loads to have actually succeeded.
- D. This condition ignores outcome entirely and runs the task after both dependencies finish, so it would still trigger reconciliation even if one or both loads failed.
Checkpoint 2 of 7· Put it in order
Put the steps for editing a task's dependencies and its condition in order
- 1.Select one of the conditional options in the Run if dependencies field
- 2.In the Depends on field, remove upstream tasks or add them from the drop-down menu
- 3.Select a task
- 4.Click Save task
Choose the upstream tasks first, then the condition that is evaluated against them, then save.
“Select one of the conditional options in the Run if dependencies field.”Source: docs.databricks.com
Sources1
2.The six Run if conditions
Depends on decides *after which* tasks a task runs. Run if decides *whether* it runs, based on whether those upstream tasks succeeded, failed or finished. The default is All succeeded. The other conditions are what you use for fallback, alerting and cleanup paths.
| Run if condition | Task runs when | State if unmet |
|---|---|---|
| All succeeded (default) | All dependencies have run and succeeded | Upstream failed |
| At least one succeeded | At least one dependency succeeded | Upstream failed |
| None failed | No dependency failed and at least one dependency ran | Upstream failed |
| All done | All dependencies have run, whatever their status | Not applicable |
| At least one failed | At least one dependency failed | Excluded |
| All failed | All dependencies failed | Excluded |
Checkpoint 3 of 7· Match them up
Match each Run if condition to the scenario it fits
Tap a term, then the definition that fits it.
All done ignores outcomes. At least one failed waits for a failure. None failed lets skipped tasks through but not failed ones. All succeeded is the strict default.
“The task is run after all its dependencies have run, regardless of the status of the dependent runs.”Source: docs.databricks.com
Sources1
3.Excluded, failed and canceled upstream tasks
Yes. When Run if conditions are evaluated, an upstream task in the Excluded state counts as successful. Tasks marked Upstream failed or Upstream canceled count as failed, not as skipped.
There is one exception. If *every* dependency of a task is excluded, the task is excluded too, whatever its Run if condition. That includes All done, which otherwise ignores upstream outcomes. This cascades down a chain: in A → B → C, excluding A excludes B, which excludes C.
Cancellation also flows downstream. If you cancel a task run, the cancellation propagates through downstream tasks. Tasks whose Run if condition handles failure still run, which means a cleanup task still runs after a cancel.
Checkpoint 4 of 7· Check yourself
In a linear chain A → B → C, task A is Excluded. C uses the All done condition. What happens?
All of B's dependencies are excluded, so B is excluded. That makes all of C's dependencies excluded too, which overrides C's All done condition.
“If all of a task's dependencies are excluded, the task itself is also excluded, regardless of its Run if condition.”Source: docs.databricks.com
Sources1
4.Disabled tasks and their downstream tasks
Disabling a task makes it skip at run time while keeping its configuration and run history. You might do this while debugging, or to keep a broken step from blocking a scheduled job. A disabled task finishes with a Disabled termination code. Each downstream task is then checked against its Run if condition, and if that condition can no longer be met, the downstream task is also disabled for that run. The key rule is that a disabled parent is never treated as a failure.
| Run if condition | Effect of a disabled parent | Example |
|---|---|---|
| All succeeded (default) | Does not satisfy the succeeded requirement | A (disabled) → B: B does not run |
| At least one succeeded | B runs if at least one other parent succeeded | A (disabled) and C (succeeded) → B: B runs |
| None failed | B runs if at least one parent completed without failure | A (disabled) and C (skipped) → B: B runs |
| All done | Treated as done | A (disabled) → B: B runs |
| At least one failed | Not treated as a failure | A (disabled) and C (failed) → B: B runs |
| All failed | Not treated as a failure | A (disabled) → B: B does not run |
Checkpoint 5 of 7· Exam question
A cleanup task should run after two independent ingestion tasks finish, but the cleanup logic itself is safe to execute whether the ingestions succeeded or failed. Which `Run if dependencies` value lets the cleanup task always execute once its dependencies are finished?
Correct answer: A — All done, which lets the task run once every dependency finishes regardless of whether it succeeded or failed
- A. All done runs the downstream task after every dependency finishes irrespective of outcome, which is the only listed condition that guarantees cleanup always runs once ingestion is complete.
- B. All succeeded blocks the task whenever any dependency fails, so a failed ingestion would leave cleanup unexecuted, which defeats the requirement that cleanup always runs.
- C. At least one failed only triggers the task on failure, so cleanup would never run in the common case where both ingestions succeed, which does not satisfy the requirement.
- D. None failed still depends on the outcome of dependencies rather than simply their completion, so a failed ingestion would prevent cleanup from running under this condition.
In the job definition, only tasks you disable yourself carry the flag. Downstream disablement is worked out when each run is created and isn't stored in the job settings.
Disabling is not the only way to skip a task. In Run now with different settings you can deselect a task, but that only affects that one run. Disabling skips the task on every run until you re-enable it.
Checkpoint 6 of 7· Fill the gap
Which field disables this task in the job settings?
{
"tasks": [
{
"task_key": "load_raw_data",
" ? ": true,
"notebook_task": {
"notebook_path": "/Shared/etl/load_raw_data"
}
}
]
}Setting disabled: true on a task through the Jobs REST API, CLI, SDK or a bundle disables it. Excluded and skipped are run states, not settings.
Source: docs.databricks.com5.Branching, looping and calling other jobs
Some task types exist only to shape the DAG. An If/else condition task runs one part of the graph or another depending on a boolean expression, for example running transformations only if ingestion added new data. Its condition can compare taskValues, job parameters or dynamic values using ==, !=, >, >=, < and <=. A For each task runs a nested task once for each element of an input array, and that array can be set by hand or generated at run time. A Run Job task starts another job in the workspace.
These task types rely on their upstream tasks. If the upstream task that supplies an If/else condition or a For each input array is disabled, the If/else or For each task fails. Pipeline tasks have their own limit: a pipeline runs only one update at a time, so a pipeline task inside a For each runs only one iteration at a time.
Checkpoint 7 of 7· Check yourself
You loop a pipeline task over 20 parameter sets with a For each task whose concurrency is set to 10. How many iterations run at once?
A pipeline can run only one update at a time. A pipeline task inside For each is therefore capped at one concurrent iteration, whatever concurrency the loop is configured with.
“A pipeline task wrapped in a for-each task is capped to one concurrent iteration”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.An Excluded upstream task counts as a failure, so it blocks a downstream task that uses All succeeded.Why is that wrong?
Excluded upstream tasks count as successful when Run if conditions are evaluated. Only Upstream failed and Upstream canceled count as failed.
Covered in Excluded, failed and canceled upstream tasks
2.Disabling a parent task sets off a downstream At least one failed or All failed handler.Why is that wrong?
A disabled parent is not treated as a failure. A failure handler runs only if another parent actually failed.
Covered in Disabled tasks and their downstream tasks
3.If a task's Run if condition isn't met, the task is always marked Upstream failed.Why is that wrong?
Only conditions built around success mark the task Upstream failed. At least one failed and All failed mark it Excluded, and the task is skipped.
Covered in The six Run if conditions
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/jobs/run-ifOfficial docs
“Databricks runs upstream tasks before running downstream tasks, running as many of them in parallel as possible.”
↩︎ Depends on: drawing the DAG“Depends on is only visible if the job consists of multiple tasks.”
↩︎ Depends on: drawing the DAG“your new task has a dependency configured on this task by default”
↩︎ Depends on: drawing the DAG“All dependencies have run and succeeded. This is the default setting.”
↩︎ The six Run if conditions“None of the dependencies failed, and at least one dependency was run.”
↩︎ The six Run if conditions“Upstream tasks marked Upstream failed or Upstream canceled are treated as failed, not as skipped.”
↩︎ Excluded, failed and canceled upstream tasks“If you cancel a task run, the cancellation propagates through downstream tasks”
↩︎ Excluded, failed and canceled upstream tasks“The Run Job task lets you trigger another job in your workspace.”
↩︎ Branching, looping and calling other jobs“Upstream tasks with the Excluded result state are treated as successful when evaluating Run if conditions.”
↩︎ Exam trap 1“At least one failed: At least one dependency failed. The task is marked as Excluded if the condition is unmet.”
↩︎ Exam trap 3“Select one of the conditional options in the Run if dependencies field.”
↩︎ Checkpoint“Tasks configured to handle failures are marked as Excluded if their Run if condition is not met.”
↩︎ Prediction“The task is run after all its dependencies have run, regardless of the status of the dependent runs.”
↩︎ Checkpoint“If all of a task's dependencies are excluded, the task itself is also excluded, regardless of its Run if condition.”
↩︎ Checkpoint - 2.
“A disabled parent task does not satisfy the succeeded requirement.”
↩︎ Disabled tasks and their downstream tasks“Lakeflow Jobs determines downstream disablement at run creation time and doesn't persist it in the job settings.”
↩︎ Disabled tasks and their downstream tasks“An If/else condition task fails if the upstream task that provides its condition value is disabled.”
↩︎ Branching, looping and calling other jobs“A disabled parent task is not treated as a failure.”
↩︎ Exam trap 2 - 3.https://docs.databricks.com/aws/en/jobs/run-nowOfficial docs
“Deselecting a task only affects this run. To skip a task on every run without deleting it, disable the task in the job settings.”
↩︎ Disabled tasks and their downstream tasks - 4.https://docs.databricks.com/aws/en/jobs/control-flowOfficial docs
“You can define If/else conditions against taskValues, job parameters, or dynamic values.”
↩︎ Branching, looping and calling other jobs
Also cited
“A pipeline task wrapped in a for-each task is capped to one concurrent iteration”
↩︎ Checkpoint