What you will be able to do
- Explain how a Lakeflow job is built from one or more tasks arranged in a task graph
- Configure a notebook task, including its source, compute and parameters
- Choose between the SQL task's query, alert and file options and meet its warehouse requirement
- Configure a dashboard task and predict whose credentials refresh the dashboard
- Add a pipeline task and predict whether it runs one update or runs continuously
Key concept
Job as a task graph (DAG) — A Lakeflow job is a set of typed tasks, such as notebook, SQL, dashboard or pipeline tasks. The dependencies between those tasks form a directed acyclic graph, and that graph decides the order in which the tasks run.
1.A job is a graph of typed tasks
Every Lakeflow job has one or more tasks, and you create a job by configuring its first task. To find a job's tasks in the workspace UI, click Jobs & Pipelines in the sidebar, open the job, and click the Tasks tab. The task graph appears there. Clicking a task name opens its configuration below the graph. If the job already has tasks, you add more with Add task.
Each task has a type. The type decides which asset the task runs and which settings the UI shows. The task-type list is long: Python script, Python wheel, dbt, JAR, Spark Submit, Run Job, If/else, For each and more. This exam objective focuses on four everyday types: Notebook, SQL, Dashboards and Pipeline. Each task also needs compute to run its logic. With serverless, Databricks configures that compute for you.
The type list also shows a separate SQL Alert (Public Preview) type. That is not the same thing as the Alert option inside the SQL task, which is the one this lesson covers.
| Task type | What it runs | Compute | Distinctive settings |
|---|---|---|---|
| Notebook | A notebook from Workspace or a Git provider | A cluster, serverless or other supported compute; a SQL warehouse only for an all-SQL notebook | Parameters read with dbutils.widgets |
| SQL | A SQL query, a SQL alert, or a .sql file | Serverless or pro SQL warehouse | Parameters (alerts do not support parameters) |
| Dashboards | A refresh of a published dashboard | Serverless or pro SQL warehouse; defaults to the dashboard's default warehouse | Subscribers, Dashboard filters |
| Pipeline | An update of an existing Lakeflow pipeline | Managed in the pipeline definition | Full refresh, parameter overrides |
Tasks are wired together through dependencies. In a task's Depends on field you pick the upstream tasks it waits for, and the lines between tasks in the job DAG show those dependencies. Databricks runs upstream tasks before downstream tasks, and runs as many of them in parallel as possible. If you have a task selected in the DAG when you create a new task, the new task depends on the selected one by default.
Checkpoint 1 of 6· Exam question
A team builds a Lakeflow Job with three tasks: `ingest_raw`, `transform_silver`, and `publish_gold`. `transform_silver` must only start after `ingest_raw` finishes, and `publish_gold` must only start after `transform_silver` finishes. How should the engineer wire this ordering into the job's task graph?
Correct answer: A — Set the `Depends on` field of `transform_silver` to `ingest_raw`, and the `Depends on` field of `publish_gold` to `transform_silver`, chaining each task to its upstream task
- A. Lakeflow Jobs builds its DAG from explicit `Depends on` links between tasks, so chaining each task to the one it must follow is exactly how execution order and dependency structure are configured.
- B. Task creation order in the UI has no bearing on run order; without explicit dependency links, independent tasks are eligible to start at the same time regardless of how they were added.
- C. Cluster assignment controls compute, not scheduling; multiple tasks can share a cluster and still run in parallel or in an order that has nothing to do with cluster configuration.
- D. Maximum concurrent runs limits how many runs of the whole job can execute at once, not the order of tasks within a single run, so it does not enforce ordering between tasks.
Checkpoint 2 of 6· Check yourself
A colleague wants to create a brand-new Lakeflow job in the workspace UI. Where do they start?
In the UI you create a job by configuring its first task. A schedule is optional, and the DAG grows as you add tasks.
“You create a new job in the workspace UI by configuring the first task.”Source: docs.databricks.com
2.Notebook tasks
To set up a notebook task, add a task, give it a name, and choose Notebook under Type. Then pick a Source. Workspace lets you browse to a notebook in the workspace, including one in a Databricks Git folder. Git provider points at a remote repository, and Databricks recommends it for versioning assets that jobs run on a schedule. With a Git provider you enter a *relative* path such as etl/bronze/ingest.py, without a leading / or ./. Only one remote Git repository can serve all the tasks in a job.
Parameters are key-value pairs that the notebook reads with dbutils.widgets. Output has limits: if one cell's output goes over 8MB, or all cells together go over 30MB, the run is canceled and marked failed.
No. A SQL warehouse can run a notebook task only if the notebook is written entirely in SQL and SQL is its default language. A notebook that mixes languages needs a cluster or other supported compute. On serverless compute, install libraries inside the notebook with the Environment pane or %pip install. On other compute, add them under Dependent libraries.
Checkpoint 3 of 6· Check yourself
Which notebook can use a SQL warehouse as the compute for its notebook task?
Both conditions must hold: every cell is SQL, and SQL is the default language. One Python cell rules out a SQL warehouse.
“as the compute for a notebook task only when the notebook is written entirely in SQL”Source: docs.databricks.com
Sources3
3.SQL tasks: query, alert or file
A notebook task carries a lot of setup that a plain SQL step doesn't need. The SQL task type runs SQL assets directly, and it requires Databricks SQL plus a serverless or pro SQL warehouse. After you choose SQL under Type, the SQL task drop-down offers three options:
- Query: runs a SQL query against the warehouse you specify.
- Alert: evaluates a SQL alert on that warehouse. You can add Subscribers to receive the results.
- File: runs a .sql file, which can contain several statements separated by semicolons. The file's source can be Workspace or a Git provider, and the same relative-path rules apply as for notebooks.
You can set optional key-value Parameters that the SQL asset references. Alerts are the exception: they don't take parameters.
Checkpoint 4 of 6· Check yourself
You configure a SQL task with the Alert option and try to pass a key-value parameter to it. What happens?
Queries and SQL files can reference task parameters. Alerts cannot.
“Alerts do not support parameters.”Source: docs.databricks.com
Sources4
4.Dashboard tasks
A dashboard task refreshes a published Databricks dashboard as one step of a job. The usual pattern is to place it downstream of the tasks that load the dashboard's data. The dashboard must be published, and you need at least CAN VIEW access to it. Choose the dashboard and, optionally, a SQL warehouse. If you leave the warehouse empty, the task uses the dashboard's default warehouse, and in either case it must be a serverless or pro warehouse. Subscribers get an email when the refresh finishes. This list is separate from any subscribers set up in the Dashboards UI.
Dashboard filters set filter values before the refresh. You can't set dataset parameters directly in the task. Attach the parameter to a filter widget on the dashboard canvas, then set that filter's value in the task.
There is also a shortcut for job-level parameters. If a dashboard filter's URL Identifier matches a job-level parameter key, the task automatically applies that parameter's value to the filter.
That depends on how the dashboard was published. With shared data permissions, the refresh always uses the publisher's credentials, and the job's Run as setting can't change that. Without shared data permissions, the job's Run as identity refreshes the dashboard. Run as defaults to the job owner, and viewers can't see results produced under that identity.
Checkpoint 5 of 6· Check yourself
A dashboard uses a dataset parameter, and you want the job to set that parameter's value on each refresh. How?
The task can only set filters. A dataset parameter has to be exposed through a filter widget first.
“Dataset parameters can't be set directly in the task.”Source: docs.databricks.com
Sources5
5.Pipeline tasks
Jobs and pipelines define relationships in different ways. A job links tasks procedurally, while a Lakeflow pipeline declares how datasets and transformations relate. A pipeline task lets a job run a pipeline. Choose Pipeline as the type and pick an existing pipeline. You can also request a full refresh and set parameter overrides. The pipeline's own definition already holds its source code and compute settings, so the task doesn't configure either.
Parameter overrides work like this: you can access job or task parameters in your pipeline with dynamic value references, and you override them by adding key-value pairs in the task's Parameters field.
How the task runs depends on the job's schedule. In a triggered or scheduled job, the pipeline task starts a single update and stops when that update completes. In a continuous job, the pipeline task runs the pipeline continuously, even if the pipeline's own Pipeline mode is triggered.
Databricks recommends running continuous pipelines through a continuous job rather than the pipeline's built-in continuous setting. In that setup, leave the pipeline's mode at triggered, which is the default. The bundle below defines that kind of job: continuous.pause_status: UNPAUSED keeps it running, and the single task points at the pipeline through pipeline_task.
# resources/continuous_job.yml
resources:
jobs:
continuous_pipeline_job:
name: continuous_pipeline_job
performance_target: STANDARD
continuous:
pause_status: UNPAUSED
email_notifications:
on_failure:
- your_email@example.com
tasks:
- task_key: refresh_pipeline
pipeline_task:
pipeline_id: ${resources.pipelines.example_pipeline.id}Pipeline tasks also come with concurrency caps, because a pipeline can only run one update at a time. A job with max_concurrent_runs above 1 that contains a pipeline task is capped to a single concurrent run. A pipeline task wrapped in a For each task is capped to one concurrent iteration, whatever concurrency the loop is configured with.
Checkpoint 6 of 6· Check yourself
You wrap a pipeline in a continuous job. What should the pipeline's own Pipeline mode be?
The job controls continuous execution. Keeping the pipeline's mode at triggered avoids unexpected behavior.
“set the pipeline's Pipeline mode to triggered (the default) when you wrap it in a continuous job”Source: docs.databricks.com
Sources6
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A notebook whose default language is SQL can always run on a SQL warehouse.Why is that wrong?
The notebook must also be written entirely in SQL. If it mixes languages, it needs a cluster or other supported compute.
Covered in Notebook tasks
2.A pipeline task follows the pipeline's own Pipeline mode setting.Why is that wrong?
The job's schedule decides the mode. A continuous job runs the pipeline continuously, and a triggered or scheduled job runs one update.
Covered in Pipeline tasks
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“A job has one or more tasks.”
↩︎ A job is a graph of typed tasks“If you are using serverless, Databricks configures your compute resources.”
↩︎ A job is a graph of typed tasks“SQL Alert (Public Preview)”
↩︎ A job is a graph of typed tasks“You create a new job in the workspace UI by configuring the first task.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/jobs/run-ifOfficial docs
“Databricks runs upstream tasks before running downstream tasks, running as many of them in parallel as possible.”
↩︎ A job is a graph of typed tasks“Configuring task dependencies creates a Directed Acyclic Graph (DAG) of task execution”
↩︎ Key concept - 3.
“select a cluster or other supported compute instead”
↩︎ A job is a graph of typed tasks“Only one remote Git repository can be used for all tasks in a job.”
↩︎ Notebook tasks“Configure Parameters as key-value pairs that can be accessed in the notebook using dbutils.widgets”
↩︎ Notebook tasks“the run is canceled and marked as failed”
↩︎ Notebook tasks“If the notebook uses another default language or mixes languages, select a cluster or other supported compute instead.”
↩︎ Exam trap 1“as the compute for a notebook task only when the notebook is written entirely in SQL”
↩︎ Checkpoint - 4.https://docs.databricks.com/aws/en/jobs/tasks/sqlOfficial docs
“The SQL task requires Databricks SQL and a serverless or pro SQL warehouse.”
↩︎ SQL tasks: query, alert or file“The file can contain multiple SQL statements separated by semicolons (;).”
↩︎ SQL tasks: query, alert or file“Alerts do not support parameters.”
↩︎ Checkpoint - 5.
“You must have at least CAN VIEW access to the dashboard you want to deploy.”
↩︎ Dashboard tasks“must use a serverless or pro SQL warehouse for dashboard tasks”
↩︎ Dashboard tasks“Dashboards with shared data permissions are always updated using the publisher's credentials.”
↩︎ Dashboard tasks“If a dashboard filter's URL Identifier matches a job-level parameter key, the task automatically applies that parameter's value to the filter.”
↩︎ Dashboard tasks“Dataset parameters can't be set directly in the task.”
↩︎ Checkpoint - 6.
“In a triggered or scheduled job, the pipeline task starts a single update and stops when the update completes.”
↩︎ Pipeline tasks“Lakeflow pipelines manage all configurations for source code and compute in the pipeline definition.”
↩︎ Pipeline tasks“You can override parameters by adding Key-Value pairs in the Parameters of your task configuration.”
↩︎ Pipeline tasks“A job with max_concurrent_runs > 1 that contains a pipeline task is capped to a single concurrent run.”
↩︎ Pipeline tasks“the job's schedule determines the execution mode and takes precedence over the Pipeline mode setting”
↩︎ Exam trap 2“the pipeline runs continuously even if its own Pipeline mode setting is triggered”
↩︎ Prediction“set the pipeline's Pipeline mode to triggered (the default) when you wrap it in a continuous job”
↩︎ Checkpoint