What you will be able to do
- Pick the Runs tab and time range that show a job's duration trend over the 60 days of retained history
- Read the matrix view's Run total duration and task rows, where bar height shows duration and colour shows status
- Turn a duration baseline into an expected completion time that raises a warning and a slow-run event
- Compare a slow run with an earlier fast run to separate a real regression from a larger data volume
- Split a slow run into its Queued, Waiting for resources, Library installation and Running phases
- Compute mean, p90 and p95 run-duration baselines from the system.lakeflow timeline tables
Key concept
Duration baseline — A baseline is the run time a job normally takes, learned from its own run history. A single run only counts as slow when you compare it with that baseline. Once you know the baseline, you can make it the job's expected completion time so that Databricks flags any run that goes past it.
1.Choosing the Runs tab and time range for a trend
A 45-minute run tells you nothing by itself. It is only alarming if the same job usually finishes in 10. Spotting a performance trend means putting the current run next to enough earlier runs to see what normal looks like. Each run's status and the Runs tab itself are covered in Job run statuses and the Runs tab. This section is about which view and which time range give you a long enough horizon.
There are two Runs tabs, and they answer different questions. The workspace-level tab under Jobs & Pipelines lists runs of every job and pipeline you can access. Its Finished runs count graph is built for spotting failures across the workspace, not one job's duration. By default it covers only the last 48 hours, and it appears only when you filter to Jobs or Pipelines. Click a job's name and you get that job's own Runs tab, with a matrix view and a list view of its active and completed runs. Trend reading happens there.
| View | What it shows | Use it for |
|---|---|---|
| Workspace Runs tab (Jobs & Pipelines > Runs) | Finished runs count graph (last 48 hours by default), Top 5 error types, and a table of runs from the last 60 days across jobs and pipelines | Seeing which jobs fail or error most across the workspace |
| A job's Runs tab (click the job name) | Matrix view and list view of that job's active and completed runs, broken down by task | Reading one job's duration trend and spotting the task whose duration changed |
The horizon is capped at 60 days. Databricks keeps run history for 60 days for both jobs and pipelines, so a filter wider than that brings back no older runs. If you need a longer baseline, export run results before they expire, or query the system tables described below. The time filter matches completed runs by their end time and in-progress runs by their start time. A run that started late on the last day of your range but finished after it falls outside the range.
Checkpoint 1 of 9· Check yourself
A data engineer wants to compare tonight's slow run of a nightly job against a 90-day baseline of its run times. What should they do?
Run history is kept for 60 days, so a 90-day filter returns no older runs in either Runs tab or either view. A longer baseline needs exported results or the system tables.
“Databricks retains run history for 60 days for both jobs and pipelines.”Source: docs.databricks.com
Sources1
2.Reading the matrix: bar height is duration
The matrix view is the run history of a single job, laid out so you can scan for trends. Each run is a column. The top row, Run total duration, shows the whole run. Below it, the Tasks rows have one cell per task. Hover over a bar to see the run's start time, duration and status. Hover over a task cell to also see the cluster it ran on.
Each bar encodes two separate things. Colour is status: green for successful, red for unsuccessful, pink for skipped, yellow for waiting for retry, and grey for pending, canceled or timed out. Height is duration, and nothing else. A taller bar means a run or task took longer. It does not mean the run cost more, was retried more often, or triggered more downstream tasks.
Because height shows duration, a regression looks like a step in the bars. Suppose a nightly job's bars stay level for three weeks and then jump. The Run total duration row tells you the whole run got slower. Scan down the task rows for the same run and you can see which task's bar grew while the others stayed the same. If only one task's bar is taller, the slowdown is in that task, not spread across the job.
Height shows duration, so that task really did take longer. If the total stayed the same, the extra time must have been offset elsewhere, for example by other tasks running in parallel or finishing faster. Hover over the task cell to see its exact start time and duration.
Checkpoint 2 of 9· Check yourself
For three weeks, a nightly job's Run total duration bars in the matrix view were level. For the last four runs they are clearly taller, and all of them are still green. In those runs, the ingest task's bar has grown, while the transform and publish task bars look the same as before. What does the matrix tell you?
Bar height shows only duration. When the total bar steps up and just one task's bar grows, that task is where the extra time went. Retries would show up as a yellow colour, not a taller bar, and cost isn't drawn in the matrix at all.
“The height of the individual job run and task run bars visually indicates the run duration.”Source: docs.databricks.com
Checkpoint 3 of 9· Exam question
In the Lakeflow Jobs Matrix view, the height of a colored bar in the "Run total duration" row and in each task row is scaled to represent which quantity for that run or task?
Correct answer: A — The bar height encodes how long that run or task took to complete, so a taller bar always represents a longer measured duration.
- A. This is correct because the Matrix view scales each cell's bar height to the measured run or task duration, letting engineers scan a row visually to spot which executions ran long. Color separately encodes status, while height is reserved for the time dimension.
- B. This is incorrect because the Matrix view does not surface billed compute cost as a visual dimension; cost is not one of the metrics the run history view renders per cell. Bar height in this view is tied to duration, not spend.
- C. This is incorrect because retry counts are not represented through bar height; a run waiting for retry is instead shown through the yellow status color, not through a taller or shorter bar. Height stays tied to elapsed duration regardless of retry activity.
- D. This is incorrect because the number of downstream tasks in a job's dependency graph is a structural property shown by the task rows themselves, not by scaling any single cell's bar height. Height in the Matrix view reflects time elapsed, not graph fan-out.
Sources1
3.Turning a baseline into a duration warning
Reading the matrix every morning doesn't scale. Instead, write the baseline into the job as an expected completion time. In the Job details pane, click Duration and streaming backlog thresholds, choose Run duration as the metric, and enter a duration in the Warning field. That value is the job's expected completion time.
After that, the job flags slow runs on its own. When a run exceeds the threshold, both the matrix view and the runs list show a warning, and an event fires that you can use to send a notification that the job is running slowly. This is how a team gets told about a nightly job running well past its baseline without anyone checking the history first. The Timeout field in the same dialog is different: it sets a maximum completion time, after which the run's status becomes Timed Out.
| Field | What it sets | What happens when exceeded |
|---|---|---|
| Warning | The job's expected completion time | A warning appears in the matrix and runs list, and an event is triggered that can send a slow-job notification |
| Timeout | The job's maximum completion time | Databricks sets the run's status to Timed Out |
Checkpoint 4 of 9· Check yourself
Your team wants to hear about it when a nightly job runs well past its usual 10 minutes, without anyone opening the run history first. Slow runs should still be allowed to finish. Which setting does this?
The Warning field sets the job's expected completion time. Runs that go past it are flagged in the matrix and runs list, and an event fires that can send a notification. Timeout stops the run and marks it Timed Out, and changing views or filters only affects what someone sees when they look.
“If you have configured an expected completion time, the matrix view displays a warning when the duration of a run exceeds the configured time.”Source: docs.databricks.com
Checkpoint 5 of 9· Exam question
A team wants to be proactively alerted whenever their nightly Lakeflow job runs noticeably longer than its historical baseline, without anyone manually checking the run history view each morning. Which configuration achieves this?
Correct answer: A — Configure an expected completion time on the job so Databricks flags a warning whenever a run's duration exceeds that threshold.
- A. This is correct because setting an expected completion time on the job establishes a duration threshold, and Databricks automatically flags and warns on any run that exceeds it, giving the team a proactive signal without manual daily checks.
- B. This is incorrect because switching to List view only changes how existing run data is displayed for manual inspection; it introduces no threshold and generates no automatic warning when a run runs long.
- C. This is incorrect because widening the start-time filter only changes which historical runs are visible when someone chooses to look; it does not create any automatic flag or notification on future slow runs.
- D. This is incorrect because filtering by run-as user narrows which runs appear in the list but has no effect on detecting or flagging duration regressions against a historical baseline.
4.Comparing a slow run with an earlier fast run
A duration warning tells you a run was slow, not why. The next step is to compare it with a recent run that met the baseline. Per-run views are covered in Single-run DAG graph and timeline views. Here, the point is to open both runs and compare task durations. The timeline view picks out tasks that took a long time. The list view lets you sort tasks by duration, so the task whose duration grew most rises to the top in the slow run.
Not necessarily. More input data makes a run take longer without anything being wrong. For serverless jobs, each task run shows rows read, rows written and the total number of queries. Open the slow run next to the earlier fast run and compare these numbers task by task:
- Rows read and rows written show whether the data volume changed.
- Total number of queries shows whether the shape of the workload changed.
- Performance insights (the lightbulb) point to inefficiencies introduced since the previous run.
If the slow run read several times as many rows, the extra time may just reflect the extra data, and the baseline may need to move. If the row counts are similar but the task still took longer, expand it in the timeline to see its individual queries. Open any query marked with a lightbulb to read its performance insights. These metrics exist only for serverless Lakeflow Jobs. On classic compute, you compare by task durations alone.
Checkpoint 6 of 9· Match them up
You have a slow serverless run and an earlier fast run of the same job open next to each other. Match each thing you compare to the question it answers.
Tap a term, then the definition that fits it.
Row counts show changes in data volume, so they tell you whether more data explains the slowdown. Query counts show changes in workload shape, and the lightbulb points to inefficiencies introduced since the fast run. Sorting by duration in the list view brings the task whose duration grew most to the top.
“Rows read and rows written to identify changes in data volume.”Source: docs.databricks.com
Checkpoint 7 of 9· Exam question
A nightly Lakeflow job that normally finishes in about 10 minutes took 45 minutes last night, and no code or configuration changed. What is the best way to determine whether this is a genuine performance regression rather than simply a larger data volume that night?
Correct answer: A — Open the slow run alongside a recent fast run and compare rows read and written plus total query counts per task to see if workload size grew.
- A. This is correct because comparing rows read and written and per-task query counts between the slow run and a recent fast run reveals whether the extra time is explained by more data flowing through the pipeline or by a genuine slowdown at the same workload size. This side-by-side diagnostic is the recommended way to separate a workload-shape change from a real regression.
- B. This is incorrect because removing the expected completion time setting only stops future warnings from appearing; it does not explain why last night's specific run took four times longer and provides no diagnostic evidence about data volume or regression.
- C. This is incorrect because changing the cluster type affects startup and warm-up behavior going forward but does not, by itself, tell you whether last night's slowdown came from more data or a genuine regression. It changes future runs rather than diagnosing the one that already happened.
- D. This is incorrect because adding retries addresses transient task failures, not diagnosis of a slow-but-successful run, and it does nothing to distinguish a data volume increase from an actual performance regression.
5.Breaking a slow run into queue, setup and execution
A run can get longer without its code running any slower. The extra time can come from waiting to start or from preparing compute. On the job run details page, hover over the run's Duration to see the total split into phases. The four phases are Queued, Waiting for resources, Library installation and Running. You can see this breakdown for both job runs and task runs.
| Phase | Part of the run | What it tells you |
|---|---|---|
| Queued | Queue | How long the run waited for concurrency limits, such as the job's maximum concurrent runs or the workspace-wide limit |
| Waiting for resources | Compute setup | How long it took to get compute before the work could start |
| Library installation | Compute setup | How long it took to install libraries before the work could start |
| Running | Execution | How long the work itself ran |
The phase that grew points to the fix. A longer Queued phase is not a code problem. One cause is that the job hit its maximum concurrent runs limit because the previous run of the same job was still going. Another is that other jobs running at the same time filled the workspace-level concurrent run limit. If Running grew instead, the task itself got slower, and the run comparison above applies. For a multi-task job, the job-level breakdown counts Waiting for resources only until the first task starts. To see each task's own setup time, open the breakdown in that task run's output.
Pipeline tasks go through different phases: created, waiting for resources, initializing, setting up tables and running. Read their breakdown with those names.
Checkpoint 8 of 9· Check yourself
Last night's run of a multi-task job took 30 minutes longer than its baseline. Its code and input data haven't changed. How do you find out whether the extra time went to waiting or to the work itself?
The phase breakdown shows where a run's time went. If Queued or setup grew, the task code didn't get slower. If Running grew, the task itself got slower. Rows read and written show changes in data volume, not waiting time. A matrix bar's colour shows status, not phases.
“To see where time is spent in a run and what you can do to reduce it, view the phase breakdown.”Source: docs.databricks.com
Sources3
6.Computing baselines across jobs with system.lakeflow
The matrix shows one job at a time and only 60 days of runs. To set baselines for many jobs at once, query the jobs system tables. With access to the system.lakeflow schema, you can query job run and task run records from across your account. You can also build dashboards on them to track duration trends.
Pick the table carefully. job_run_timeline has phase columns such as queue_duration_seconds, setup_duration_seconds and execution_duration_seconds. They are filled in only for single-task jobs created with the legacy Jobs APIs and are 0 for every other run. For multi-task jobs, use system.lakeflow.job_task_run_timeline. It reports durations per task run, keyed by task_key and job_run_id. Long runs are sliced into hourly periods across several rows, which is why baseline queries add up period_end_time - period_start_time for each run.
with job_run_duration as (
SELECT
workspace_id,
job_id,
run_id,
CAST(SUM(period_end_time - period_start_time) AS LONG) as duration
FROM
system.lakeflow.job_run_timeline
WHERE
period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
GROUP BY ALL
)SELECT
t1.workspace_id,
t1.job_id,
COUNT(DISTINCT t1.run_id) as runs,
MEAN(t1.duration) as mean_seconds,
AVG(t1.duration) as avg_seconds,
PERCENTILE(t1.duration, 0.9) as p90_seconds,
PERCENTILE(t1.duration, 0.95) as p95_seconds
FROM
job_run_duration t1
GROUP BY ALL
ORDER BY mean_seconds DESC
LIMIT 100The two values do different jobs. The mean is the typical run. The p90 and p95 columns show how long the job's longest runs take. That makes the p95 a sensible starting point for the Warning field: normal busy runs stay under it, and a real regression goes over it. You can also put the same query behind a SQL alert to catch duration anomalies across many jobs at once.
Checkpoint 9 of 9· Fill the gap
In system.lakeflow.job_run_timeline, a long job run is sliced into hourly periods stored across several rows. Which aggregate turns those rows into one total duration per run in this query?
with job_run_duration as (
SELECT
workspace_id,
job_id,
run_id,
CAST( ? (period_end_time - period_start_time) AS LONG) as duration
FROM
system.lakeflow.job_run_timeline
WHERE
period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
GROUP BY ALL
)A long run is split into hourly periods across several rows, so you get its full duration by adding up every period's length. MAX would return only the longest slice, AVG the average slice length and COUNT the number of slices. None of them is the run's total time.
Source: docs.databricks.comSources
(5)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/jobs/monitorOfficial docs
“Databricks retains run history for 60 days for both jobs and pipelines.”
↩︎ Choosing the Runs tab and time range for a trend“The selected time range is matched against each run's end time for completed runs, or its start time for in-progress runs.”
↩︎ Choosing the Runs tab and time range for a trend“The Finished runs count graph shows the number of runs completed in the last 48 hours.”
↩︎ Choosing the Runs tab and time range for a trend“The Finished runs count graph appears only when you filter to Jobs or Pipelines.”
↩︎ Choosing the Runs tab and time range for a trend“If you need to preserve job runs, Databricks recommends exporting results before they expire.”
↩︎ Choosing the Runs tab and time range for a trend“The matrix view shows a history of runs for the job, including each job task.”
↩︎ Reading the matrix: bar height is duration“The height of the individual job run and task run bars visually indicates the run duration.”
↩︎ Reading the matrix: bar height is duration“Successful runs are green. Unsuccessful runs are red, skipped runs are pink, and waiting for retry is yellow.”
↩︎ Reading the matrix: bar height is duration“If you have configured an expected completion time, the matrix view displays a warning when the duration of a run exceeds the configured time.”
↩︎ Turning a baseline into a duration warning“You can search for a task by name, filter by task status or task type, and sort tasks by status, name, or duration.”
↩︎ Comparing a slow run with an earlier fast run“Jobs have a timeline view to identify tasks that are taking a long time to complete”
↩︎ Comparing a slow run with an earlier fast run“If you have access to the system.lakeflow schema, you can also view and query records of job runs and tasks from across your account.”
↩︎ Computing baselines across jobs with system.lakeflow“If you have configured an expected completion time, the matrix view displays a warning when the duration of a run exceeds the configured time.”
↩︎ Key concept“The height of the individual job run and task run bars visually indicates the run duration.”
↩︎ Exam trap 1“Databricks retains run history for 60 days for both jobs and pipelines.”
↩︎ Exam trap 3“To see where time is spent in a run and what you can do to reduce it, view the phase breakdown.”
↩︎ Checkpoint - 2.
“Enter a duration in the Warning field to configure the job's expected completion time.”
↩︎ Turning a baseline into a duration warning“If the job exceeds this threshold, an event is triggered. You can use this event to notify when a job is running slowly.”
↩︎ Turning a baseline into a duration warning“To configure a maximum completion time for a job, enter the maximum duration in the Timeout field.”
↩︎ Turning a baseline into a duration warning“If the job exceeds this threshold, an event is triggered. You can use this event to notify when a job is running slowly.”
↩︎ Exam trap 2 - 3.
“These same metrics make it simple to spot differences between a slow run and a previous fast run.”
↩︎ Comparing a slow run with an earlier fast run“If higher data volume explains the increased duration, the slowdown might be expected.”
↩︎ Comparing a slow run with an earlier fast run“These metrics and insights apply to serverless Lakeflow Jobs only. Job runs on classic compute don't show this information.”
↩︎ Comparing a slow run with an earlier fast run“hover over the run's Duration to see the total duration broken down into the following phases”
↩︎ Breaking a slow run into queue, setup and execution“For a job with multiple tasks, the job-level breakdown counts Waiting for resources only until the first task starts running.”
↩︎ Breaking a slow run into queue, setup and execution“The job reached its maximum concurrent runs limit because a previous run of the same job is still in progress.”
↩︎ Breaking a slow run into queue, setup and execution“Pipeline tasks have different phases than most tasks.”
↩︎ Breaking a slow run into queue, setup and execution“If higher data volume explains the increased duration, the slowdown might be expected.”
↩︎ Exam trap 4“Rows read and rows written to identify changes in data volume.”
↩︎ Checkpoint - 4.
“For all non-legacy or multi-task job runs, use the job_task_run_timeline table, which reports durations per task run.”
↩︎ Computing baselines across jobs with system.lakeflow“For non-legacy or multi-task job runs, these columns log a value of 0.”
↩︎ Computing baselines across jobs with system.lakeflow“For details on how Databricks slices long runs into hourly intervals, see timeline slicing logic.”
↩︎ Computing baselines across jobs with system.lakeflow“A top 90 and a 95 percentile column show the average lengths of the job's longest runs.”
↩︎ Computing baselines across jobs with system.lakeflow“For non-legacy or multi-task job runs, these columns log a value of 0.”
↩︎ Exam trap 5 - 5.https://docs.databricks.com/aws/en/lakehouse-architecture/deployment-guide/observabilityOfficial docs
“Anomaly detection: Set up SQL alerts to monitor job duration anomalies or repeated failures.”
↩︎ Computing baselines across jobs with system.lakeflow