What you will be able to do
- Find the failed task in a job run using the Runs tab matrix view and the Task run details page
- Tell a failed task apart from a skipped one, and use error codes, logs and query profiles to narrow down the cause
- Pick the right fix for a configuration, compute, concurrency or upstream failure before you re-run anything
Key concept
Repair run — A repair re-runs only the tasks of a multi-task job run that did not succeed, plus the tasks that depend on them. Successful work is not repeated. Because the repair uses the current settings, you diagnose and fix the cause first, then repair.
1.Finding the failed task in the Runs tab
A failure notice can come from an email notification, a monitoring tool or the Lakeflow Jobs UI. Whichever way it arrives, you start in the same place. Click Jobs & Pipelines in the sidebar, then click the job's name in the Name column. This opens the job's Runs tab, which lists active and completed runs.
The Runs tab has a matrix view: one column per run and one row per task. It is the fastest way to find which task broke. A task can be unsuccessful in two ways. Either it failed, or it was skipped because a task it depends on failed. So when you see several unsuccessful cells in one column, the task that actually broke is usually the earliest one. The cells after it are often just its downstream tasks being skipped.
Hover over a failed cell to see its metadata: start and end dates, status, duration, cluster details and sometimes an error message. To see more, click the failed task. The Task run details page opens and shows the task's output, its error message and its metadata.
Checkpoint 1 of 4· Put it in order
Put these steps in the order you follow to reach the cause of a failed task.
- 1.Click Jobs & Pipelines in the sidebar
- 2.Use the matrix view to spot the unsuccessful task cell and hover over it for metadata
- 3.Click the failed task to open the Task run details page
- 4.Click the job name in the Name column to open its Runs tab
You go from the job list to the job's Runs tab, find the red cell in the matrix, and then click it. The Task run details page shows the output and the error message.
“To help identify the cause of the failure, click the failed task.”Source: docs.databricks.com
Sources1
2.Reading the colours, error codes and run details
The bars in the matrix are colour-coded, and the colour tells you what kind of problem you have before you open anything. The height of each bar shows how long the run took. If you set an expected completion time, the matrix also warns you when a run goes over it.
| Colour | Status | What to do with it |
|---|---|---|
| Green | Successful | Nothing; a repair won't re-run it unless it depends on a task being repaired |
| Red | Unsuccessful | Open its Task run details; this is usually where the cause is |
| Pink | Skipped | Look upstream for the failed task it depends on |
| Yellow | Waiting for retry | A retry is pending; the run isn't finished yet |
| Grey | Pending, canceled or timed out | Check whether the run was stopped or hit a timeout |
Sometimes you need a wider view than one job, for example when you suspect the same fault is hitting many jobs. The workspace-level Runs tab under Jobs & Pipelines lists runs from all jobs and pipelines you can access. You can filter it by Error code for failed runs. It also has a Top 5 error types table that shows the most common error types in the selected time range. Each row in the list shows the error code the run ended with and the run's parameters. The parameters column is useful when a bad parameter value is the suspect.
For a single run, the job run details page shows the job output and links to logs. Click a task node in the graph to see its source code, the cluster that ran it with links to that cluster's query history and logs, and metrics for the task. For serverless jobs, queries and query profiles are built into the timeline view. Click a statement there to open its query profile.
Checkpoint 2 of 4· Check yourself
Several jobs in your workspace started failing this morning and you want to know which error is the most common. Where do you look?
The workspace Runs tab ranks the most frequent error types across jobs for the selected time range. The other options only show one job or one task at a time.
“The Top 5 error types table shows a list of the most frequent error types from the selected time range”Source: docs.databricks.com
Sources2
3.Matching the cause to a fix
Tasks fail for many reasons, such as a data quality issue, a misconfiguration or not enough compute. Each cause has its own fix, and you apply that fix before re-running anything.
| Cause | Remediation |
|---|---|
| Task configuration is wrong | Click Edit task, update the configuration in the new tab, click Save task |
| Insufficient cluster resources | Under Compute click Configure to change workers or instance types, click Swap to switch to another available cluster, consider a shared all-purpose cluster instead of a job cluster, or ask an admin to raise cloud quotas |
| Maximum concurrent runs exceeded | Wait for other runs to complete, or use Edit concurrent runs to raise Maximum concurrent runs |
| Upstream problem, e.g. external data source unavailable | Wait until the external issue is resolved, then use repair run |
The last row matters. When the cause is outside your job, nothing in the job needs to change. You still use the repair feature, but only after the external source is back.
You also have some help with diagnosis. Genie Code helps diagnose job errors: open the failed job from the Jobs UI and select Diagnose Error.
Continuous jobs behave differently. When consecutive failures of a continuous job pass a threshold, Lakeflow Jobs retries the job with exponential backoff. The Job details pane then shows three things: the number of consecutive failures, how long the job must run without error to count as successful, and the time until the next retry. Once you have fixed the cause, click Restart run. This cancels the active run, resets the retry period and starts a new run, so you don't have to wait for the next backoff retry.
Checkpoint 3 of 4· Match them up
Match each failure situation to its remediation.
Tap a term, then the definition that fits it.
Each cause has its own fix: configuration problems are fixed in the task, compute problems in the cluster, concurrency problems in the job's concurrency limit, and a continuous job stuck in backoff is restarted with Restart run.
“To cancel the active run, reset the retry period, and start a new job run, click Restart run.”Source: docs.databricks.com
Checkpoint 4 of 4· Exam question
A Lakeflow Job has two tasks, `ingest_raw` and `transform_silver`, where `transform_silver` depends on `ingest_raw`. In today's run, `ingest_raw` succeeded but `transform_silver` failed because it referenced an incorrect `source_path` parameter value. The team wants to fix the parameter and get `transform_silver` running again without spending compute rerunning `ingest_raw`. What should they do?
Correct answer: A — Open the Repair job run dialog, enter the corrected `source_path` value for the failed task, and confirm; the unsuccessful task re-executes.
- A. Repairing a run re-executes only the tasks that did not succeed, along with any tasks that depend on them, while leaving already-successful upstream tasks untouched; parameters entered in the dialog override the values used for the repaired attempt.
- B. Changing the job-level default and starting a fresh run re-executes every task, including the already-successful upstream task, which wastes the compute the team is trying to avoid spending.
- C. Cloning the job and discarding the failed run's history throws away the audit trail of what happened and still requires a full run of the new job definition rather than a targeted repair of the one failed task.
- D. Starting over with `Run now` re-executes the entire task graph from the beginning, which reruns the task that already completed successfully instead of only the one that failed.
Sources1
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Every unsuccessful cell in the matrix view is a task whose own code failed, so each one needs investigating.Why is that wrong?
A task can also be unsuccessful because it was skipped after a task it depends on failed. Find the task that actually failed upstream; the skipped tasks after it don't have their own bug.
Covered in Finding the failed task in the Runs tab
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“The matrix view in the Runs tab shows a history of runs for the job, including successful and unsuccessful runs for each job task.”
↩︎ Finding the failed task in the Runs tab“The Task run details page appears, displaying the task's output, error message, and associated metadata.”
↩︎ Finding the failed task in the Runs tab“If the failure is related to the task configuration, click Edit task.”
↩︎ Matching the cause to a fix“You can still take advantage of the repair run feature covered in the next section after the external issue is resolved.”
↩︎ Matching the cause to a fix“When consecutive failures of a continuous job exceed a threshold, Lakeflow Jobs uses exponential backoff to retry the job.”
↩︎ Matching the cause to a fix“Genie Code helps diagnose errors in jobs.”
↩︎ Matching the cause to a fix“repair failed or canceled multi-task jobs by running only the subset of unsuccessful tasks and any dependent tasks”
↩︎ Key concept“A task run may be unsuccessful because it failed or was skipped because a dependent task failed.”
↩︎ Exam trap 1“To help identify the cause of the failure, click the failed task.”
↩︎ Checkpoint“To cancel the active run, reset the retry period, and start a new job run, click Restart run.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/jobs/monitorOfficial docs
“The cluster that ran the task and links to its query history and logs.”
↩︎ Reading the colours, error codes and run details“For serverless jobs, queries and query profiles are integrated into the timeline view.”
↩︎ Reading the colours, error codes and run details“Successful runs are green. Unsuccessful runs are red, skipped runs are pink, and waiting for retry is yellow.”
↩︎ Prediction“The Top 5 error types table shows a list of the most frequent error types from the selected time range”
↩︎ Checkpoint