What you will be able to do
- Build an If/else condition task from job parameters, task parameters or task values
- Predict how If/else operators compare numeric and string operands
- Wire true and false branches to downstream tasks
- Configure a For each task with inputs, concurrency and a nested task, and read its iteration outputs downstream
1.If/else condition tasks: branching on a value
A Run if condition branches on whether upstream tasks *succeeded or failed*. The If/else condition task type branches on a *value* instead. It evaluates a boolean expression made of two operands and one operator, and the result decides which part of the DAG runs. The documentation's example is a process_records task that counts invalid rows in a task value called bad_records. An If/else task can then send the run to a separate path whenever bad records exist:
{{tasks.process_records.values.bad_records}} > 0An operand can reference a job parameter ({{job.parameters.<name>}}), a task parameter, or a task value ({{tasks.<task_name>.values.<value_name>}}). The supported operators are ==, !=, >, >=, < and <=. Task values are set in Python notebooks with dbutils.jobs.taskValues.set(). Databricks recommends reading them through the dynamic value reference pattern shown above.
dbutils.jobs.taskValues.set(key = "record_count", value = query.count())
dbutils.jobs.taskValues.set(key = "booking_status", value = query.take(1)[0][0])How a value is compared depends on the operator, not on what the value looks like. == and != compare their operands as strings. >, >=, < and <= compare them as numbers, so 12.0 >= 12 is true and 10.0 >= 12 is false. When an operand references a task value, only numeric, string and boolean values are allowed, and any other type makes the condition fail. Non-numeric values are serialized to strings, so a boolean task value becomes "true" or "false".
Checkpoint 1 of 5· Exam question
A Lakeflow Job has an ingestion task followed by a cleanup task that deletes staging files. The team wants the cleanup task to run every time the job executes, whether the ingestion task succeeds, fails, or is skipped. Which Run if condition should be set on the cleanup task's dependency on the ingestion task?
Correct answer: D — All done
- A. This condition only runs the downstream task when the ingestion task succeeded, so a failed ingestion run would skip the cleanup step entirely. That contradicts the requirement that cleanup always executes regardless of outcome.
- B. This condition is meant for multiple upstream dependencies and still requires at least one of them to succeed before the downstream task runs. With a single ingestion dependency that fails, the cleanup task would still be skipped.
- C. This condition blocks the downstream task whenever a dependency actually failed, since a failure counts against it. That means cleanup would not run on a failed ingestion run, which is exactly the case that must be covered.
- D. This condition runs the downstream task after its dependencies have run, regardless of whether they succeeded, failed, or were skipped. That matches the requirement that the cleanup task execute on every job run no matter the ingestion outcome.
- E. This condition only triggers when a failure occurred upstream, so on a normal successful ingestion run the cleanup task would be skipped. That leaves the staging files undeleted on the majority of successful runs.
Checkpoint 2 of 5· Check yourself
An upstream notebook sets a task value is_valid to the Python boolean True. Which If/else condition correctly tests it?
Booleans are serialized to the string "true" or "false", and == does a string comparison, so the operand must be compared with lowercase true.
“For example, if a task value is set to a boolean value, it is serialized to "true" or "false".”Source: docs.databricks.com
2.Building the task and wiring its true and false branches
You create an If/else task on the Tasks tab. Click Add task and give it a name. In Type, choose If/else condition. Enter the first operand, pick a boolean operator, enter the second operand, and save. Retries and notifications are available under advanced settings, the same as for other task types.
Checkpoint 3 of 5· Put it in order
Put the steps for adding an If/else condition task in order
- 1.Enter a name into the Task name field
- 2.Enter the operand to be evaluated in the first Condition text box
- 3.Select a boolean operator, then enter the comparison value in the second Condition text box
- 4.Click Add task
- 5.In the Type drop-down menu, select If/else condition
The UI flow goes from creating the task, to naming it, to choosing its type, to building the expression from left operand, operator and right operand.
“In the Type drop-down menu, select If/else condition.”Source: docs.databricks.com
To attach branches, select the If/else task in the graph and click Add task. The new task's Depends on field defaults to <task-name> (true). Change it to <task-name> (false) for a task that should run when the condition is false. Each outcome can lead to several tasks, run in serial or in parallel. Branches are about values; if a branch should depend on upstream *failures*, combine it with Run if dependencies.
There is one dependency failure mode to remember. If the upstream task that supplies the condition value is disabled, the If/else task itself fails.
A disabled task is skipped, so the task value is never produced. An If/else task fails when the upstream task providing its condition value is disabled, so every run of the job would fail at the branch.
Sources2
3.For each tasks: looping over inputs
The For each task adds looping. It runs a nested task once per element of an input array and passes a different element to each iteration. A typical use is running the same transformations over many tables, one table name per iteration. Configuring it takes two tasks: the For each task itself and the nested task, which can be any standard task type *except* another For each task.
Inputs go in the Inputs box as a JSON array. The elements can be strings, numbers, booleans, key-value pairs or arbitrarily complex objects. They can also come from a task value or a job parameter reference. Inside the nested task's parameters, {{input}} gives the current element and {{input.<key>}} reads a field of an object element. Concurrency sets how many iterations run in parallel; the default is 1. Nested iterations that don't depend on each other can run concurrently.
[{"table": "sales"}, {"table": "orders"}]Checkpoint 4 of 5· Fill the gap
An upstream notebook generates the list of countries a For each task will loop over. Which method completes this call?
dbutils.jobs.taskValues. ? (key = "countries", value = countries_array)The upstream task publishes the array with taskValues.set. The For each Inputs box then references it as {{tasks.generate_countries_list.values.countries}}.
Source: docs.databricks.com| Input method | Limit |
|---|---|
| JSON typed directly into the Inputs text box | 5,000 characters |
| Task value reference, e.g. {{tasks.<task_name>.values.<task_value_name>}} | 48 KB for the referenced value |
| Job parameter reference, e.g. {{job.parameters.<name>}} | 10,000 characters for the parameter value |
If your parameters need more than 48 KB, pass lookup keys instead of the full data. The documented example keeps per-table arguments in a JSON config file, sets the Inputs to a short list of keys such as ["table_1","table_2","table_3"], and has the nested notebook look up its arguments with {{input}}. Because those steps don't depend on each other, the example sets concurrency above 1.
4.Depending on a loop and collecting its results
In the DAG, the For each task is the top-level node. Downstream tasks depend on the For each task and cannot name the nested task as a dependency. Each iteration can still set task values, and a downstream task can read all of them as one list in iteration order. To do that, reference the *nested* task's key, not the For each task's key. Reading these aggregated outputs is in Beta. It is enabled by default and requires Databricks Runtime 15.4 LTS or above.
results = dbutils.jobs.taskValues.get(taskKey = "process", key = "result")
# results == [1, 2, 3]The dynamic value reference {{tasks.process.values.result}} resolves to the same array. An iteration that didn't set the key keeps its slot as a null, so a missing second value gives [1, None, 3]. As with If/else, a For each task fails if the upstream task that supplies its input values is disabled. In the run history, a For each task appears as a table of its iterations.
Checkpoint 5 of 5· Check yourself
A For each task named loop_tables wraps a nested notebook task named process. A downstream report task needs every iteration's result value. What should it do?
Downstream tasks can only depend on the top-level For each task, and aggregated values are read with the nested task's key.
“use the key of the nested task that runs inside the For each task, not the For each task itself”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.An If/else condition
12.0 == 12evaluates to true because the numbers are equal.Why is that wrong?== and != compare operands as strings, so it is false. Only >, >=, < and <= compare numerically.
2.You can nest a For each task inside another For each task to loop over two dimensions.Why is that wrong?
The nested task must be one of the standard task types, and another For each task is explicitly not allowed.
Covered in For each tasks: looping over inputs
3.A downstream task can set the nested task inside a For each as its dependency.Why is that wrong?
Only the top-level For each task can be a dependency. Downstream tasks reach iteration outputs through the nested task's task values instead.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/jobs/control-flowOfficial docs
“You can define If/else conditions against taskValues, job parameters, or dynamic values.”
↩︎ If/else condition tasks: branching on a value - 2.
“Only numeric, string, and boolean values are allowed when referencing task values in an operand.”
↩︎ If/else condition tasks: branching on a value“Select <task-name> (false) to configure a task that runs on a false condition evaluation.”
↩︎ Building the task and wiring its true and false branches“An If/else condition task fails if the upstream task that provides its condition value is disabled.”
↩︎ Building the task and wiring its true and false branches“The >, >=, <, and <= operators perform numeric comparisons of their operands.”
↩︎ Exam trap 1“The == and != operators perform string comparison of their operands. For example, 12.0 == 12 evaluates to false.”
↩︎ Prediction“For example, if a task value is set to a boolean value, it is serialized to "true" or "false".”
↩︎ Checkpoint“In the Type drop-down menu, select If/else condition.”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/jobs/task-valuesOfficial docs
“Databricks recommends referencing task values as task parameters configured using the dynamic value reference pattern {{tasks.<task_name>.values.<value_name>}}.”
↩︎ If/else condition tasks: branching on a value“An iteration that did not set the key keeps its position in the result”
↩︎ Depending on a loop and collecting its results“use the key of the nested task that runs inside the For each task, not the For each task itself”
↩︎ Checkpoint - 4.
“enter a Concurrency value for the task. The default value is 1.”
↩︎ For each tasks: looping over inputs“If your parameters require more than 48 KB, you can pass a lookup to a larger config file.”
↩︎ For each tasks: looping over inputs“A For each task fails if the upstream task that provides its input values is disabled.”
↩︎ Depending on a loop and collecting its results“You cannot add another For each task as the nested task.”
↩︎ Exam trap 2“A downstream task cannot set the nested task as a dependency directly”
↩︎ Exam trap 3 - 5.
“Nested tasks can use the key to retrieve the specific data needed for each iteration.”
↩︎ For each tasks: looping over inputs