What you will be able to do
- Explain how jobs, tasks and triggers fit together in Lakeflow Jobs
- Create a job with a notebook task in the Jobs & Pipelines UI
- Choose a trigger type and know what concurrency and pausing do to runs
- Configure retries, timeouts and disabled tasks, and know the serverless default for retries
Key concept
Job (Lakeflow Jobs) — A job is the named, schedulable resource that wraps one or more tasks arranged as a DAG. A trigger decides when it runs, and job-level settings such as parameters, notifications and Git apply to all of its tasks.
1.Jobs, tasks and triggers
Lakeflow Jobs is the orchestration layer of Databricks. An ETL workload might ingest raw files, check them for nulls, transform them, and run a quality check. In Lakeflow Jobs, that is one job made of several tasks. The documentation names three main concepts: jobs, tasks and triggers. A job holds the tasks, and Databricks draws them as a Directed Acyclic Graph (DAG). A job can be one notebook or hundreds of tasks with dependencies and conditional logic. Job-level properties include the trigger, run-time parameters that are pushed to every task, failure or slow-run notifications, and Git source-control settings.
| Concept | What it is | Examples from the docs |
|---|---|---|
| Job | The resource that coordinates, schedules and runs tasks, shown as a DAG | A daily job that ingests revenue data and branches on a null check |
| Task | A specific unit of work inside a job | Notebook task, pipeline task, Python script task |
| Trigger | What starts a job run, based on time or on an event | Every day at 2 AM; when new data arrives in cloud storage |
Every job needs four things: a task that holds the logic, a compute resource to run it (serverless, classic jobs compute or all-purpose compute), a unique name, and a schedule. The schedule is the one you can skip. A job without a schedule only runs when someone starts it.
Checkpoint 1 of 5· Check yourself
A team wants a job that analysts start by hand after a monthly close. Which statement about the job's schedule is correct?
The schedule is the one job requirement that is optional. Without one, the job runs only when someone starts it manually.
“Optionally, you can omit setting a schedule and trigger the job manually.”Source: docs.databricks.com
Sources1
2.Creating an ETL job in the Jobs & Pipelines UI
In the UI, you don't create an empty job. You create a job by configuring its first task. Open Jobs & Pipelines in the sidebar, click Create and then Job, and pick the Notebook tile. If that tile isn't shown, use Add another task type. Give the task a name, choose a notebook for Path, and click Save task. The job appears in the jobs list as New Job <date> <time>, and you can rename it later. If the workspace doesn't have serverless compute for jobs, you also have to choose compute. For that case, Databricks recommends jobs compute.
Checkpoint 2 of 5· Put it in order
Put the steps for creating a job with a notebook task in the UI in order
- 1.Click Create, then Job
- 2.Click Save task
- 3.Click the Notebook tile to configure the first task
- 4.Enter a Task name and select a notebook for the Path field
- 5.Click Jobs & Pipelines in the sidebar
The job comes into existence when its first task is saved, so all the task configuration happens inside the Create flow.
“Jobs contain one or more tasks. You create a new job by configuring the first task for that job.”Source: docs.databricks.com
In an ETL job, later tasks are added with Add task in the task graph on the Tasks tab. Each task can depend on upstream tasks. The UI offers many task types, including Notebook, Python script, Python wheel, SQL, Pipeline, Ingestion pipeline, dbt, JAR, Spark Submit and Run Job. It also has the control-flow types If/else and For each. Branching and looping have a lesson of their own. Here, the point is that they are task types you add from the same graph. The UI also connects to automation. From a job's menu you can Switch to code version (YAML), and the job page shows the JSON definition that the REST API uses.
Checkpoint 3 of 5· Exam question
A Lakeflow Job has a `send_notification` task that depends on the upstream `validate_data` and `transform_data` tasks. The team wants `send_notification` to run and report the job outcome after both upstream tasks finish, whether they succeeded or failed. Which `run_if` condition should be configured on `send_notification`'s dependency on the two upstream tasks?
Correct answer: A — Configure `run_if: ALL_DONE` on the dependency so the task executes once both upstream tasks have finished running, independent of whether either task succeeded or failed.
- A. This is the correct setting: it makes the dependent task run once its dependencies have completed regardless of their outcome, which is exactly what an unconditional notification task needs.
- B. This is the default dependency behavior, but it means the notification task is skipped entirely whenever either upstream task fails, defeating the goal of always reporting the outcome.
- C. This condition only triggers the task when a failure occurred, so a fully successful run would never produce a notification, which does not match the requirement.
- D. This condition still skips the task whenever an upstream failure occurs, so it cannot guarantee a notification is sent for every possible outcome.
Sources2
3.Triggers: deciding when the job runs
A new job runs only when you start it. To automate it, open the job and go to Schedules & Triggers in the Job details pane. Click Add trigger, choose a type, and save. From then on, the job runs whenever that trigger fires.
| Trigger type | When a run starts |
|---|---|
| Scheduled | On a time-based schedule |
| Table update | When source tables are updated |
| File arrival | When new files arrive in a monitored Unity Catalog storage location |
| Model update | When a Unity Catalog model is created, a model version is ready, or a model alias is set (Beta) |
| Continuous | Whenever the previous run completes or fails, so the job is always running |
| None (manual) | Run now button or an external orchestration tool |
Checkpoint 4 of 5· Match them up
Match each ETL requirement to the trigger type that fits it
Tap a term, then the definition that fits it.
Each trigger type responds to a different event. File arrival watches a Unity Catalog storage location, Table update watches source tables, Scheduled follows the clock, and Continuous starts the next run as soon as the last one ends.
“Triggers a job run when new files arrive in a monitored Unity Catalog storage location.”Source: docs.databricks.com
By default, only one run of a job can be active at a time. You can raise this limit in Advanced settings, which matters if overlapping runs are acceptable for your ETL. You can also pause a trigger. A paused trigger stops starting new runs but leaves in-flight runs alone. Resume brings back the same schedule. If a run is still active when you resume a continuous trigger, the scheduler waits for that run to finish before starting another.
No. Pausing only stops the trigger from starting new runs. The active run continues to completion.
Sources3
4.Job settings, retries and timeouts
The Job details side pane is where you set the schedule or trigger, job parameters, compute, tags, notifications, maximum concurrent runs, duration thresholds and Git settings. Job parameters are passed to tasks that accept key-value parameters. Tags propagate to the job clusters a run creates, so they also show up in cluster monitoring. Databricks recommends tags only for non-sensitive values. Under Duration and streaming backlog thresholds, a Warning duration raises an event you can use for a notification. A Timeout marks the job Timed Out if it doesn't finish in time.
Retries are set per task under Retries. The default depends on the job configuration. For most configurations, failed tasks are not retried. Serverless jobs are the exception, because they auto-optimize retries by default. The retry interval is measured in milliseconds from the start of the failed run. If you set both a timeout and retries, the timeout applies to each retry, not to the whole sequence. In the job's JSON, these settings appear as task fields:
"depends_on": [
{
"task_key": "success"
}
],
"disable_auto_optimization": true,
"email_notifications": {},
"max_retries": 3,
"min_retry_interval_millis": 300000,Checkpoint 5 of 5· Check yourself
A task has a 30-minute Timeout and 2 Retries. How does the timeout apply?
The timeout is applied to each attempt separately, so every retry gets the full 30 minutes.
“If you configure both Timeout and Retries, the timeout applies to each retry.”Source: docs.databricks.com
There are two ways to handle a broken task without taking the job apart. Disable a task to skip it at run time but keep its configuration and run history. This is useful while you debug an upstream issue or keep the rest of the job on schedule. Clone a task or a whole job to copy its configuration. A cloned job is identical except for its job ID.
Sources4
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Every job must have a schedule before it can run.Why is that wrong?
The schedule is optional. A job without one runs only when triggered manually, from the UI or programmatically.
Covered in Jobs, tasks and triggers
2.Tasks are never retried unless you configure Retries yourself.Why is that wrong?
That is true for most configurations, but serverless jobs auto-optimize retries by default.
Covered in Job settings, retries and timeouts
3.If a scheduled run fires while the previous run is still going, the two runs execute in parallel.Why is that wrong?
By default, only one run can be active at a time. Runs beyond the configured maximum concurrency are skipped.
Covered in Triggers: deciding when the job runs
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/jobsOfficial docs
“There are three main concepts when using Lakeflow Jobs for orchestration in Databricks: jobs, tasks, and triggers.”
↩︎ Jobs, tasks and triggers“A job is the primary resource for coordinating, scheduling, and running your operations.”
↩︎ Key concept - 2.
“Databricks recommends always using jobs compute when configuring tasks.”
↩︎ Creating an ETL job in the Jobs & Pipelines UI“You can also view the JSON definitions for use with REST API get, create, and reset endpoints.”
↩︎ Creating an ETL job in the Jobs & Pipelines UI“Optionally, you can omit setting a schedule and trigger the job manually.”
↩︎ Exam trap 1“Optionally, you can omit setting a schedule and trigger the job manually.”
↩︎ Checkpoint“Jobs contain one or more tasks. You create a new job by configuring the first task for that job.”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/jobs/triggersOfficial docs
“Any currently active runs continue, but the trigger no longer starts new runs.”
↩︎ Triggers: deciding when the job runs“By default, only a single run of a job can be active at a time.”
↩︎ Exam trap 3“Triggers a job run when new files arrive in a monitored Unity Catalog storage location.”
↩︎ Checkpoint“Runs are skipped when they exceed the configured max concurrency for a job.”
↩︎ Prediction - 4.
“For most configurations, the default setting does not retry any tasks on task failure.”
↩︎ Job settings, retries and timeouts“Disable a task to skip it at run time without removing it from the job.”
↩︎ Job settings, retries and timeouts“Serverless jobs auto-optimize retries by default.”
↩︎ Exam trap 2“If you configure both Timeout and Retries, the timeout applies to each retry.”
↩︎ Checkpoint