CertSafari
    Databricks Certified Data Engineer Associate· Lessons

    Domain 6 · Lesson 25/33

    Job run history trends and duration baselines in Lakeflow Jobs

    Identify trends in job performance using the Lakeflow Jobs run history view to compare current execution times against historical baselines

    18 min read
    2% of exam
    5 sources
    Published 22 Oct 2026
    Docs as of 17 Oct 2026

    What you will be able to do

    • Pick the Runs tab and time range that show a job's duration trend over the 60 days of retained history
    • Read the matrix view's Run total duration and task rows, where bar height shows duration and colour shows status
    • Turn a duration baseline into an expected completion time that raises a warning and a slow-run event
    • Compare a slow run with an earlier fast run to separate a real regression from a larger data volume
    • Split a slow run into its Queued, Waiting for resources, Library installation and Running phases
    • Compute mean, p90 and p95 run-duration baselines from the system.lakeflow timeline tables

    Key concept

    Duration baseline — A baseline is the run time a job normally takes, learned from its own run history. A single run only counts as slow when you compare it with that baseline. Once you know the baseline, you can make it the job's expected completion time so that Databricks flags any run that goes past it.

    1.Choosing the Runs tab and time range for a trend

    A 45-minute run tells you nothing by itself. It is only alarming if the same job usually finishes in 10. Spotting a performance trend means putting the current run next to enough earlier runs to see what normal looks like. Each run's status and the Runs tab itself are covered in Job run statuses and the Runs tab. This section is about which view and which time range give you a long enough horizon.

    There are two Runs tabs, and they answer different questions. The workspace-level tab under Jobs & Pipelines lists runs of every job and pipeline you can access. Its Finished runs count graph is built for spotting failures across the workspace, not one job's duration. By default it covers only the last 48 hours, and it appears only when you filter to Jobs or Pipelines. Click a job's name and you get that job's own Runs tab, with a matrix view and a list view of its active and completed runs. Trend reading happens there.

    Which Runs tab to open for which question
    ViewWhat it showsUse it for
    Workspace Runs tab (Jobs & Pipelines > Runs)Finished runs count graph (last 48 hours by default), Top 5 error types, and a table of runs from the last 60 days across jobs and pipelinesSeeing which jobs fail or error most across the workspace
    A job's Runs tab (click the job name)Matrix view and list view of that job's active and completed runs, broken down by taskReading one job's duration trend and spotting the task whose duration changed

    The horizon is capped at 60 days. Databricks keeps run history for 60 days for both jobs and pipelines, so a filter wider than that brings back no older runs. If you need a longer baseline, export run results before they expire, or query the system tables described below. The time filter matches completed runs by their end time and in-progress runs by their start time. A run that started late on the last day of your range but finished after it falls outside the range.

    Checkpoint 1 of 9· Check yourself

    A data engineer wants to compare tonight's slow run of a nightly job against a 90-day baseline of its run times. What should they do?

    Sources1

    2.Reading the matrix: bar height is duration

    The matrix view is the run history of a single job, laid out so you can scan for trends. Each run is a column. The top row, Run total duration, shows the whole run. Below it, the Tasks rows have one cell per task. Hover over a bar to see the run's start time, duration and status. Hover over a task cell to also see the cluster it ran on.

    Each bar encodes two separate things. Colour is status: green for successful, red for unsuccessful, pink for skipped, yellow for waiting for retry, and grey for pending, canceled or timed out. Height is duration, and nothing else. A taller bar means a run or task took longer. It does not mean the run cost more, was retried more often, or triggered more downstream tasks.

    Because height shows duration, a regression looks like a step in the bars. Suppose a nightly job's bars stay level for three weeks and then jump. The Run total duration row tells you the whole run got slower. Scan down the task rows for the same run and you can see which task's bar grew while the others stayed the same. If only one task's bar is taller, the slowdown is in that task, not spread across the job.

    Checkpoint 2 of 9· Check yourself

    For three weeks, a nightly job's Run total duration bars in the matrix view were level. For the last four runs they are clearly taller, and all of them are still green. In those runs, the ingest task's bar has grown, while the transform and publish task bars look the same as before. What does the matrix tell you?

    Checkpoint 3 of 9· Exam question

    In the Lakeflow Jobs Matrix view, the height of a colored bar in the "Run total duration" row and in each task row is scaled to represent which quantity for that run or task?

    Sources1

    3.Turning a baseline into a duration warning

    Reading the matrix every morning doesn't scale. Instead, write the baseline into the job as an expected completion time. In the Job details pane, click Duration and streaming backlog thresholds, choose Run duration as the metric, and enter a duration in the Warning field. That value is the job's expected completion time.

    After that, the job flags slow runs on its own. When a run exceeds the threshold, both the matrix view and the runs list show a warning, and an event fires that you can use to send a notification that the job is running slowly. This is how a team gets told about a nightly job running well past its baseline without anyone checking the history first. The Timeout field in the same dialog is different: it sets a maximum completion time, after which the run's status becomes Timed Out.

    Two duration thresholds on the same job
    FieldWhat it setsWhat happens when exceeded
    WarningThe job's expected completion timeA warning appears in the matrix and runs list, and an event is triggered that can send a slow-job notification
    TimeoutThe job's maximum completion timeDatabricks sets the run's status to Timed Out

    Checkpoint 4 of 9· Check yourself

    Your team wants to hear about it when a nightly job runs well past its usual 10 minutes, without anyone opening the run history first. Slow runs should still be allowed to finish. Which setting does this?

    Checkpoint 5 of 9· Exam question

    A team wants to be proactively alerted whenever their nightly Lakeflow job runs noticeably longer than its historical baseline, without anyone manually checking the run history view each morning. Which configuration achieves this?

    Sources21

    4.Comparing a slow run with an earlier fast run

    A duration warning tells you a run was slow, not why. The next step is to compare it with a recent run that met the baseline. Per-run views are covered in Single-run DAG graph and timeline views. Here, the point is to open both runs and compare task durations. The timeline view picks out tasks that took a long time. The list view lets you sort tasks by duration, so the task whose duration grew most rises to the top in the slow run.

    Not necessarily. More input data makes a run take longer without anything being wrong. For serverless jobs, each task run shows rows read, rows written and the total number of queries. Open the slow run next to the earlier fast run and compare these numbers task by task:

    • Rows read and rows written show whether the data volume changed.
    • Total number of queries shows whether the shape of the workload changed.
    • Performance insights (the lightbulb) point to inefficiencies introduced since the previous run.

    If the slow run read several times as many rows, the extra time may just reflect the extra data, and the baseline may need to move. If the row counts are similar but the task still took longer, expand it in the timeline to see its individual queries. Open any query marked with a lightbulb to read its performance insights. These metrics exist only for serverless Lakeflow Jobs. On classic compute, you compare by task durations alone.

    Checkpoint 6 of 9· Match them up

    You have a slow serverless run and an earlier fast run of the same job open next to each other. Match each thing you compare to the question it answers.

    Tap a term, then the definition that fits it.

    Checkpoint 7 of 9· Exam question

    A nightly Lakeflow job that normally finishes in about 10 minutes took 45 minutes last night, and no code or configuration changed. What is the best way to determine whether this is a genuine performance regression rather than simply a larger data volume that night?

    Sources31

    5.Breaking a slow run into queue, setup and execution

    A run can get longer without its code running any slower. The extra time can come from waiting to start or from preparing compute. On the job run details page, hover over the run's Duration to see the total split into phases. The four phases are Queued, Waiting for resources, Library installation and Running. You can see this breakdown for both job runs and task runs.

    Run phases and the question each one answers
    PhasePart of the runWhat it tells you
    QueuedQueueHow long the run waited for concurrency limits, such as the job's maximum concurrent runs or the workspace-wide limit
    Waiting for resourcesCompute setupHow long it took to get compute before the work could start
    Library installationCompute setupHow long it took to install libraries before the work could start
    RunningExecutionHow long the work itself ran

    The phase that grew points to the fix. A longer Queued phase is not a code problem. One cause is that the job hit its maximum concurrent runs limit because the previous run of the same job was still going. Another is that other jobs running at the same time filled the workspace-level concurrent run limit. If Running grew instead, the task itself got slower, and the run comparison above applies. For a multi-task job, the job-level breakdown counts Waiting for resources only until the first task starts. To see each task's own setup time, open the breakdown in that task run's output.

    Pipeline tasks go through different phases: created, waiting for resources, initializing, setting up tables and running. Read their breakdown with those names.

    Checkpoint 8 of 9· Check yourself

    Last night's run of a multi-task job took 30 minutes longer than its baseline. Its code and input data haven't changed. How do you find out whether the extra time went to waiting or to the work itself?

    Sources3

    6.Computing baselines across jobs with system.lakeflow

    The matrix shows one job at a time and only 60 days of runs. To set baselines for many jobs at once, query the jobs system tables. With access to the system.lakeflow schema, you can query job run and task run records from across your account. You can also build dashboards on them to track duration trends.

    Pick the table carefully. job_run_timeline has phase columns such as queue_duration_seconds, setup_duration_seconds and execution_duration_seconds. They are filled in only for single-task jobs created with the legacy Jobs APIs and are 0 for every other run. For multi-task jobs, use system.lakeflow.job_task_run_timeline. It reports durations per task run, keyed by task_key and job_run_id. Long runs are sliced into hourly periods across several rows, which is why baseline queries add up period_end_time - period_start_time for each run.

    Step 1: add up each run's periods into one duration per run over the last 7 dayssql
    with job_run_duration as (
        SELECT
            workspace_id,
            job_id,
            run_id,
            CAST(SUM(period_end_time - period_start_time) AS LONG) as duration
        FROM
            system.lakeflow.job_run_timeline
        WHERE
          period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
        GROUP BY ALL
    )
    Step 2: turn those durations into a baseline for each job: mean, p90 and p95 secondssql
    SELECT
        t1.workspace_id,
        t1.job_id,
        COUNT(DISTINCT t1.run_id) as runs,
        MEAN(t1.duration) as mean_seconds,
        AVG(t1.duration) as avg_seconds,
        PERCENTILE(t1.duration, 0.9) as p90_seconds,
        PERCENTILE(t1.duration, 0.95) as p95_seconds
    FROM
        job_run_duration t1
    GROUP BY ALL
    ORDER BY mean_seconds DESC
    LIMIT 100

    The two values do different jobs. The mean is the typical run. The p90 and p95 columns show how long the job's longest runs take. That makes the p95 a sensible starting point for the Warning field: normal busy runs stay under it, and a real regression goes over it. You can also put the same query behind a SQL alert to catch duration anomalies across many jobs at once.

    Checkpoint 9 of 9· Fill the gap

    In system.lakeflow.job_run_timeline, a long job run is sliced into hourly periods stored across several rows. Which aggregate turns those rows into one total duration per run in this query?

    with job_run_duration as (
        SELECT
            workspace_id,
            job_id,
            run_id,
            CAST( ? (period_end_time - period_start_time) AS LONG) as duration
        FROM
            system.lakeflow.job_run_timeline
        WHERE
          period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
        GROUP BY ALL
    )

    Sources145

    Sources

    (5)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Databricks retains run history for 60 days for both jobs and pipelines.”
      ↩︎ Choosing the Runs tab and time range for a trend
      “The selected time range is matched against each run's end time for completed runs, or its start time for in-progress runs.”
      ↩︎ Choosing the Runs tab and time range for a trend
      “The Finished runs count graph shows the number of runs completed in the last 48 hours.”
      ↩︎ Choosing the Runs tab and time range for a trend
      “The Finished runs count graph appears only when you filter to Jobs or Pipelines.”
      ↩︎ Choosing the Runs tab and time range for a trend
      “If you need to preserve job runs, Databricks recommends exporting results before they expire.”
      ↩︎ Choosing the Runs tab and time range for a trend
      “The matrix view shows a history of runs for the job, including each job task.”
      ↩︎ Reading the matrix: bar height is duration
      “The height of the individual job run and task run bars visually indicates the run duration.”
      ↩︎ Reading the matrix: bar height is duration
      “Successful runs are green. Unsuccessful runs are red, skipped runs are pink, and waiting for retry is yellow.”
      ↩︎ Reading the matrix: bar height is duration
      “If you have configured an expected completion time, the matrix view displays a warning when the duration of a run exceeds the configured time.”
      ↩︎ Turning a baseline into a duration warning
      “You can search for a task by name, filter by task status or task type, and sort tasks by status, name, or duration.”
      ↩︎ Comparing a slow run with an earlier fast run
      “Jobs have a timeline view to identify tasks that are taking a long time to complete”
      ↩︎ Comparing a slow run with an earlier fast run
      “If you have access to the system.lakeflow schema, you can also view and query records of job runs and tasks from across your account.”
      ↩︎ Computing baselines across jobs with system.lakeflow
      “If you have configured an expected completion time, the matrix view displays a warning when the duration of a run exceeds the configured time.”
      ↩︎ Key concept
      “The height of the individual job run and task run bars visually indicates the run duration.”
      ↩︎ Exam trap 1
      “Databricks retains run history for 60 days for both jobs and pipelines.”
      ↩︎ Exam trap 3
      “To see where time is spent in a run and what you can do to reduce it, view the phase breakdown.”
      ↩︎ Checkpoint
    2. 2.
      “Enter a duration in the Warning field to configure the job's expected completion time.”
      ↩︎ Turning a baseline into a duration warning
      “If the job exceeds this threshold, an event is triggered. You can use this event to notify when a job is running slowly.”
      ↩︎ Turning a baseline into a duration warning
      “To configure a maximum completion time for a job, enter the maximum duration in the Timeout field.”
      ↩︎ Turning a baseline into a duration warning
      “If the job exceeds this threshold, an event is triggered. You can use this event to notify when a job is running slowly.”
      ↩︎ Exam trap 2
    3. 3.
      “These same metrics make it simple to spot differences between a slow run and a previous fast run.”
      ↩︎ Comparing a slow run with an earlier fast run
      “If higher data volume explains the increased duration, the slowdown might be expected.”
      ↩︎ Comparing a slow run with an earlier fast run
      “These metrics and insights apply to serverless Lakeflow Jobs only. Job runs on classic compute don't show this information.”
      ↩︎ Comparing a slow run with an earlier fast run
      “hover over the run's Duration to see the total duration broken down into the following phases”
      ↩︎ Breaking a slow run into queue, setup and execution
      “For a job with multiple tasks, the job-level breakdown counts Waiting for resources only until the first task starts running.”
      ↩︎ Breaking a slow run into queue, setup and execution
      “The job reached its maximum concurrent runs limit because a previous run of the same job is still in progress.”
      ↩︎ Breaking a slow run into queue, setup and execution
      “Pipeline tasks have different phases than most tasks.”
      ↩︎ Breaking a slow run into queue, setup and execution
      “If higher data volume explains the increased duration, the slowdown might be expected.”
      ↩︎ Exam trap 4
      “Rows read and rows written to identify changes in data volume.”
      ↩︎ Checkpoint
    4. 4.
      “For all non-legacy or multi-task job runs, use the job_task_run_timeline table, which reports durations per task run.”
      ↩︎ Computing baselines across jobs with system.lakeflow
      “For non-legacy or multi-task job runs, these columns log a value of 0.”
      ↩︎ Computing baselines across jobs with system.lakeflow
      “For details on how Databricks slices long runs into hourly intervals, see timeline slicing logic.”
      ↩︎ Computing baselines across jobs with system.lakeflow
      “A top 90 and a 95 percentile column show the average lengths of the job's longest runs.”
      ↩︎ Computing baselines across jobs with system.lakeflow
      “For non-legacy or multi-task job runs, these columns log a value of 0.”
      ↩︎ Exam trap 5
    5. 5.
      “Anomaly detection: Set up SQL alerts to monitor job duration anomalies or repeated failures.”
      ↩︎ Computing baselines across jobs with system.lakeflow

    Spotted a mistake, or was something unclear? Tell us.