What you will be able to do
- Use the run breakdown to find which phase of a run took the most time
- Use the timeline view and task run history to find the task that slowed down
- Compare a slow serverless run with a faster earlier run using rows read, rows written, query count and insights
- Query system.lakeflow job tables to get duration baselines that go beyond the UI
1.Which phase took the time?
Say a job's run history shows that today's run took much longer than earlier runs. To open that run, go to the job's Runs tab and click the link in the run's Start time column. That opens the job run details page. The next question is *where* the extra time went. Total duration doesn't tell you whether the job was actually computing for longer or just waiting to start.
On the run details page, hover over the run's Duration. You see the total split into four phases: Queued, Waiting for resources, Library installation and Running. The breakdown is available for both job runs and task runs. For multi-task jobs, the job-level view counts Waiting for resources only until the first task starts, so check each task's breakdown in its task run output. Pipeline tasks use different phases: created, waiting for resources, initializing, setting up tables, and running.
The phases tell you which kind of slowdown you have. A longer Running phase means the work itself took longer. A longer Queued phase means the run was held back by concurrency limits. For example, the job may have reached its maximum concurrent runs because a previous run of the same job was still going, or the workspace may have hit its concurrent run limit. In that case the code didn't change. Only the start was delayed.
Checkpoint 1 of 5· Check yourself
A run's breakdown shows Running took the usual 12 minutes, but Queued took 18 minutes. What does this most likely mean?
The Queued phase is time spent waiting because of concurrency limits. One listed cause is the job hitting its maximum concurrent runs while an earlier run is still in progress.
“The job reached its maximum concurrent runs limit because a previous run of the same job is still in progress.”Source: docs.databricks.com
Sources1
2.Finding the task that slowed down
If the Running phase grew, look at the tasks. Jobs with several tasks have a graph view, a timeline view and a list view on the run details page. The timeline view is meant for finding tasks that take a long time to complete. It also shows dependencies and where tasks overlap, which helps you see whether one slow task held up everything after it. In the list view you can sort tasks by duration, so the slowest task comes to the top.
To check whether that task's slowdown is new, look at the task's own history. Click the task to open the Task run details page. Then pick other runs of the same task, successful and unsuccessful, from the run history drop-down menu. A For each task works the same way. You can click its node on the job run details page or its cell in the matrix view, and its runs are shown as a table of the nested task's iterations.
Checkpoint 2 of 5· Put it in order
Put these steps in order to open a single task's run history, starting from the sidebar
- 1.Click the task to open the Task run details page
- 2.Select another run of the task from the run history drop-down menu
- 3.Click Jobs & Pipelines in the sidebar
- 4.Click the run's link in the Start time column to open the job run details
- 5.Click the job's Name link to open its Runs tab
You go from the jobs list to the job's Runs tab, then into one run, then into one task. The task's history drop-down is on the Task run details page.
“Click a task on the Job run details page. The Task run details page appears.”Source: docs.databricks.com
Sources2
3.Comparing a slow run with a faster earlier run (serverless)
A longer run isn't always a problem. Sometimes the job simply had more data to process. For serverless jobs, Databricks shows some Query Profile metrics directly in the job run UI. This needs the *Improved Lakeflow Performance Observability* preview to be enabled. The metrics are rows read and rows written per task run, the total number of queries per task run, and a lightbulb indicator when a task's queries have performance insights. Open a slow run and an earlier fast run side by side, then compare:
- Rows read and rows written: did the data volume change? - Total number of queries: did the workload change shape? - Performance insights: has something inefficient appeared since the earlier run?
If more data explains the extra time, the slowdown might be expected. If it doesn't, expand the task in the timeline view and click a query that has a lightbulb to read its insights. Keep two limits in mind. These metrics exist only for serverless jobs, not for runs on classic compute. And the totals cover only the first 100 queries in a run.
Checkpoint 3 of 5· Check yourself
A serverless task now takes twice as long as it did last week. Its rows read also doubled, and no performance insights appear. What is the most reasonable conclusion?
The documentation says that when higher data volume explains the longer duration, the slowdown might be expected. You only need to look at individual queries when volume doesn't explain it.
“If higher data volume explains the increased duration, the slowdown might be expected.”Source: docs.databricks.com
Checkpoint 4 of 5· Exam question
A team wants to be proactively alerted whenever their nightly Lakeflow job runs noticeably longer than its historical baseline, without anyone manually checking the run history view each morning. Which configuration achieves this?
Correct answer: A — Configure an expected completion time on the job so Databricks flags a warning whenever a run's duration exceeds that threshold.
- A. This is correct because setting an expected completion time on the job establishes a duration threshold, and Databricks automatically flags and warns on any run that exceeds it, giving the team a proactive signal without manual daily checks.
- B. This is incorrect because switching to List view only changes how existing run data is displayed for manual inspection; it introduces no threshold and generates no automatic warning when a run runs long.
- C. This is incorrect because widening the start-time filter only changes which historical runs are visible when someone chooses to look; it does not create any automatic flag or notification on future slow runs.
- D. This is incorrect because filtering by run-as user narrows which runs appear in the list but has no effect on detecting or flagging duration regressions against a historical baseline.
Sources1
4.Baselines from system tables
The UI is good for looking at one job at a time. To get a numeric baseline, for example the typical runtime and the slowest runs for every job in the workspace, query the jobs system tables. If you have access to the system.lakeflow schema, you can query records of job runs and tasks from across your account. The documentation's sample query first works out each run's duration as the sum of period_end_time - period_start_time from system.lakeflow.job_run_timeline over the last 7 days. It then summarizes those durations per job:
SELECT
t1.workspace_id,
t1.job_id,
COUNT(DISTINCT t1.run_id) as runs,
MEAN(t1.duration) as mean_seconds,
AVG(t1.duration) as avg_seconds,
PERCENTILE(t1.duration, 0.9) as p90_seconds,
PERCENTILE(t1.duration, 0.95) as p95_seconds
FROM
job_run_duration t1
GROUP BY ALL
ORDER BY mean_seconds DESC
LIMIT 100A run that comes in above the job's p95 is clearly slower than usual. That is the same comparison you make by eye in the matrix view, but here it is a number. One detail in the schema is easy to get wrong. The duration columns in job_run_timeline (setup_duration_seconds, queue_duration_seconds, run_duration_seconds, cleanup_duration_seconds, execution_duration_seconds) are filled in only for single-task jobs created with the legacy Jobs APIs. For multi-task and non-legacy runs they hold 0. For those runs, use job_task_run_timeline, which reports durations per task run. Note that the sample queries avoid this problem because they compute duration from the period timestamps instead.
Checkpoint 5 of 5· Fill the gap
This sample returns the historical runtime of a one-time (SUBMIT_RUN) run by name. Which table completes the FROM clause?
SELECT
workspace_id,
run_id,
SUM(period_end_time - period_start_time) as run_time
FROM system.lakeflow. ?
WHERE
run_type="SUBMIT_RUN"
AND run_name = :run_name
AND period_start_time > CURRENT_TIMESTAMP() - INTERVAL 60 DAYS
GROUP BY ALLThe documentation's historical-runtime sample reads run-level periods from system.lakeflow.job_run_timeline and adds up each run's period durations.
Source: docs.databricks.comExam traps
Each one states something that sounds right. Open it to see what is actually true.
1.run_duration_seconds in system.lakeflow.job_run_timeline gives a reliable duration for every job run.Why is that wrong?
The duration columns are filled in only for single-task jobs created with the legacy Jobs APIs. Multi-task and non-legacy runs log 0, so use job_task_run_timeline for those runs.
Covered in Baselines from system tables
2.You can compare rows read, rows written and performance insights between any two job runs.Why is that wrong?
These metrics and insights are shown only for serverless Lakeflow Jobs. Runs on classic compute don't show them.
Covered in Comparing a slow run with a faster earlier run (serverless)
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“hover over the run's Duration to see the total duration broken down into the following phases:”
↩︎ Which phase took the time?“Use the breakdown to identify which phase takes the most time”
↩︎ Which phase took the time?“These same metrics make it simple to spot differences between a slow run and a previous fast run.”
↩︎ Comparing a slow run with a faster earlier run (serverless)“Databricks aggregates metrics across the first 100 queries in a job run.”
↩︎ Comparing a slow run with a faster earlier run (serverless)“These metrics and insights apply to serverless Lakeflow Jobs only. Job runs on classic compute don't show this information.”
↩︎ Exam trap 2“For a job with multiple tasks, the job-level breakdown counts Waiting for resources only until the first task starts running.”
↩︎ Prediction“The job reached its maximum concurrent runs limit because a previous run of the same job is still in progress.”
↩︎ Checkpoint“If higher data volume explains the increased duration, the slowdown might be expected.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/jobs/monitorOfficial docs
“Jobs that contain multiple tasks have a timeline view to identify tasks that are taking a long time to complete”
↩︎ Finding the task that slowed down“Select the task run in the run history drop-down menu.”
↩︎ Finding the task that slowed down“If you have access to the system.lakeflow schema, you can also view and query records of job runs and tasks from across your account.”
↩︎ Baselines from system tables“Click a task on the Job run details page. The Task run details page appears.”
↩︎ Checkpoint - 3.
“A top 90 and a 95 percentile column show the average lengths of the job's longest runs.”
↩︎ Baselines from system tables“For all non-legacy or multi-task job runs, use the job_task_run_timeline table, which reports durations per task run.”
↩︎ Exam trap 1