What you will be able to do
- Use the Spark UI to go from a job to its stages, tasks and executors to find where a workload is failing or stalling
- Take a thread dump of a hanging task or a stuck driver
- Pick driver logs, executor logs or the compute event log for a given symptom, and say which access modes let you read each one
- Use a termination error code from the compute event log to find the cause and fix of a failed cluster launch
Key concept
Drill-down from Spark UI to logs — Troubleshooting a Spark workload works like a funnel. You start at the job, move to the stage, then to the task and the executor it ran on. Then you leave the UI for the driver or executor logs, because they hold the stack traces and output that the UI does not show.
1.Reading the Spark UI: from job to task
To open the Spark UI, go to the Compute page, select your compute, and click the Spark UI tab. It shows what your application is doing while it runs. The top level is a list of jobs. Click a job to see its stages and a DAG visualization of the operations and how they depend on each other. Click a stage to see its individual tasks. The task page is the deepest the Spark UI goes. It shows how many tasks ran, which executors ran them, and shuffle information.
The docs say plainly that "Grayed boxes represent skipped stages." Structured Streaming checkpoints its state, so streaming DAGs often show skipped stages, and that is normal. On the task page, check that the tasks are spread across several executors. If one executor does all the work while others sit idle, you have a parallelism problem, even if nothing fails.
When a streaming job is running, a Streaming tab appears. It shows the input rate, the processing time and the last 1000 completed batches. Rule of thumb: "it is good if you can process each batch within 80% of your batch processing time." If processing time comes close to or goes over the batch interval, batches queue up into a backlog that can eventually take the stream down. If the Streaming tab is missing altogether, the stream probably never started. In that case, look in the driver logs (covered below).
For slow or costly workloads, Databricks has a separate Spark UI guide with a fixed order of steps. It starts with the Jobs Timeline and works down to the cause of a slow stage.
Checkpoint 1 of 6· Put it in order
Put the Spark UI cost-and-performance diagnosis steps in the order the guide gives them
- 1.Look for skew or spill
- 2.Use the Jobs Timeline to identify major issues
- 3.Determine if longest stage is I/O bound
- 4.Look at longest stage
- 5.Look for other causes of slow stage runtime
The guide says to follow these steps in order: start with the timeline, find the longest stage, check it for skew or spill, check whether it is I/O bound, and only then look for other causes.
“Use the Jobs Timeline to identify major issues”Source: docs.databricks.com
Checkpoint 2 of 6· Exam question
A data engineer runs a Spark SQL query on a SQL warehouse that joins a 900 GB fact table to a 50 GB dimension table on `region_id`. The query profile shows an Exchange operator consuming 70% of wall-clock time, with 220 GB of `spill (disk)` reported and one task running far longer than the others on that shuffle stage. Which diagnosis best matches this query profile evidence?
Correct answer: A — The shuffle is partitioned on a skewed key, so most rows land on one partition and that task spills to disk while the other tasks finish quickly.
- A. A long-tailed single task combined with heavy disk spill on the Exchange operator is the classic signature of partition skew: a small number of key values receive a disproportionate share of rows, overloading one reducer while the rest sit idle. This matches both the timing and spill evidence in the scenario.
- B. A 50 GB dimension table would typically exceed the default broadcast threshold and force a shuffle join, but that alone explains a shuffle existing, not why one specific task takes far longer and spills 220 GB while its peers do not.
- C. Disk cache affects how quickly source files are re-read from storage on subsequent scans, but the query profile evidence here points to the Exchange (shuffle) operator, not to repeated table scans, so a cache configuration would not produce this pattern.
- D. A string-typed join key can prevent some pushdown optimizations, but predicate pushdown filters rows before a scan; it does not explain a shuffle stage where one task processes far more data and spills to disk than its peers.
2.Thread dumps for hanging tasks and stuck drivers
Sometimes nothing fails and nothing finishes. For that case the Spark UI gives you thread dumps: a snapshot of the states of a JVM's threads. The docs say "Thread dumps are useful in debugging a specific hanging or slow-running task."
Getting a task's thread dump takes several steps, because you have to link a task to the executor it runs on. Open the Jobs tab, then the job, then the stage. In the stage's Tasks list, note the Task ID and Executor ID. Then go to the Executors tab, find that executor's row, and click the link in the Thread Dump column. In the dump, pick the thread whose name contains TID followed by your Task ID.
The task had already finished. Thread dumps only show live threads, and the docs note: "If the task has finished running, you will not find a matching thread."
The driver can hang too. Typical signs are no Spark progress bars at all, or progress bars stuck at 100%. For that, you don't need a task: on the Executors tab, find the driver row and click its Thread Dump link.
Checkpoint 3 of 6· Check yourself
A notebook query's progress bar has sat at 100% for 20 minutes and nothing else happens. Where is the most direct place to look in the Spark UI?
A bar stuck at 100% is one of the documented signs of a hanging driver. The driver's thread dump is on the Executors tab, in the driver row.
“In the Executors table, in the driver row, click the link in the Thread Dump column.”Source: docs.databricks.com
Sources1
3.Driver logs vs executor logs, and who can read them
The Spark UI shows you where something went wrong. The logs show you why. Driver logs hold exception stack traces. For example, if a streaming job never starts, "You can drill into the Driver logs to look at the stack trace of the exception." They also capture output from your code: "Any print statements as part of the DAG shows up in the logs too."
Executor logs are what you need when particular tasks misbehave. From the task details page, find which executor ran the task. Then on the compute UI page, click the number of nodes, then the master. From there, "You can choose the worker where the suspicious task was run and then get to the log4j output."
| Log | Standard access mode | Dedicated access mode |
|---|---|---|
| Driver logs | Workspace admins only | The dedicated user or group, and workspace admins |
| Executor logs | Not available | The dedicated user or group, and workspace admins |
Checkpoint 4 of 6· Check yourself
A workspace admin needs executor logs to investigate a failing task on Standard access mode compute. What happens?
On Standard mode, admins can read driver logs, but executor logs are not available to anyone. To get executor logs you need Dedicated access mode compute.
“Executor logs are not available for compute with Standard access mode.”Source: docs.databricks.com
Checkpoint 5 of 6· Exam question
A Structured Streaming job's micro-batches have started taking roughly eight times longer than usual. In the Spark UI, the engineer opens the Stages tab for the slow batch and finds a stage with a long tail: 199 of 200 tasks finish in under 10 seconds, but one task runs for over 20 minutes and shows a much larger `Shuffle Read Size` than the others. Which of the following are appropriate next steps to confirm and address this using the Spark UI? (Select all that apply)(Select 3)
Correct answers: A, B, C — Open the Executors tab to check whether the executor running the slow task is under GC pauses or memory pressure that would compound the skew.; Open the associated SQL/DataFrame tab to inspect the physical plan and confirm which operator produced the oversized shuffle partition.; Enable adaptive query execution's skew join optimization, or increase `spark.sql.shuffle.partitions`, so the oversized partition splits across more tasks.
- A. Checking the executor running the outlier task for GC pauses or memory pressure is a standard Spark UI workflow: a struggling executor can turn ordinary skew into an extreme long tail, and this rules that factor in or out before changing the job.
- B. The physical plan in the SQL/DataFrame tab identifies exactly which shuffle-producing operator (such as a join or aggregation) created the oversized partition, which is necessary before deciding how to rewrite or reconfigure the query.
- C. Adaptive query execution's skew join handling, or a higher shuffle partition count, directly targets an oversized shuffle partition by splitting it into smaller pieces so no single task carries a disproportionate share of the data.
- D. Driver memory pressure affects planning and coordination, not the size of an individual executor task's shuffle partition, so a larger driver node would not address a single skewed reducer task.
- E. Structured Streaming micro-batches run as ordinary Spark jobs and expose the same Stages tab with per-task shuffle metrics as batch jobs, so this claim about batch-only visibility is incorrect.
- F. Speculative execution reruns slow tasks on other executors as a mitigation, but disabling it removes that safety net without doing anything to rebalance the skewed partition itself, so the underlying skew remains.
Sources1
4.Compute event logs and termination error codes
Some failures happen before Spark even starts. The compute won't launch, or it terminates by itself. The Spark UI has nothing to show in that case. The compute event log is where you look instead. Databricks frames its troubleshooting guidance around this: "Use the error code from your cluster event log to find the relevant cause and recommended fix." For example, if a new workspace's first compute doesn't respond, the advice is: "After approximately 20-30 minutes, check your compute event log." There you may find a network-misconfiguration message. If the metastore is not working, look for METASTORE_DOWN events in your compute event logs.
What matters for the exam is reading the error code to tell *your* configuration problems apart from transient provider problems:
| What the event log reports | What it means | Recommended response |
|---|---|---|
| Not enough free addresses in subnet | The subnet has too few free IP addresses | Use a different or auto availability zone, expand the CIDR range, or clean up unused network resources |
| Insufficient capacity for the instance type in the zone | AWS has no capacity for that instance type there | Change the zone or instance type, use on-demand instead of spot, or enable flexible node types |
| INSTANCE_INITIALIZING ... threshold 700 seconds timed out | VM bootstrap timed out (network, artifact downloads or provider) | Check connectivity to artifact storage and the control plane, DNS, and firewall rules |
| CLOUD_PROVIDER_LAUNCH_FAILURE | The cloud provider failed to launch the VM | Try again later, since most of these are transient |
| CLUSTER_OPERATION_TIMEOUT (SERVICE_FAULT) | An internal operation missed its deadline (Databricks or provider side) | Retry; if it keeps happening, contact support with the cluster ID and timestamp |
Checkpoint 6 of 6· Match them up
Match each symptom to the diagnostic source you should open first
Tap a term, then the definition that fits it.
Launch failures are recorded in the event log. Startup exceptions go to the driver logs. Logs for a single task are on its executor's worker. A live task that hangs is diagnosed with a thread dump.
“Use the error code from your cluster event log to find the relevant cause and recommended fix.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A grayed-out stage in the Spark UI DAG means the stage failed.Why is that wrong?
Grayed boxes are skipped stages. Spark didn't need to recompute them because the data was already checkpointed or cached.
Covered in Reading the Spark UI: from job to task
2.Workspace admins can always read executor logs, whatever the access mode.Why is that wrong?
On Standard access mode, executor logs are not available at all. Admins can read driver logs there, but not executor logs.
Covered in Driver logs vs executor logs, and who can read them
3.A CLUSTER_OPERATION_TIMEOUT termination means your workload or cluster configuration is wrong.Why is that wrong?
This code points to a problem on the Databricks or cloud-provider side. It is usually transient and clears when you retry.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“it is good if you can process each batch within 80% of your batch processing time.”
↩︎ Reading the Spark UI: from job to task“Thread dumps are useful in debugging a specific hanging or slow-running task.”
↩︎ Thread dumps for hanging tasks and stuck drivers“If the task has finished running, you will not find a matching thread”
↩︎ Thread dumps for hanging tasks and stuck drivers“For compute with Standard access mode, only workspace admins can access driver logs.”
↩︎ Driver logs vs executor logs, and who can read them“You can choose the worker where the suspicious task was run and then get to the log4j output.”
↩︎ Driver logs vs executor logs, and who can read them“This is the most granular level of debugging you can get into from the Spark UI for a Spark application.”
↩︎ Key concept“Grayed boxes represent skipped stages.”
↩︎ Exam trap 1“Executor logs are not available for compute with Standard access mode.”
↩︎ Exam trap 2“If the data is checkpointed or cached, then Spark skips recomputing those stages.”
↩︎ Prediction“In the Executors table, in the driver row, click the link in the Thread Dump column.”
↩︎ Checkpoint“Executor logs are not available for compute with Standard access mode.”
↩︎ Checkpoint - 2.
“Look for skew or spill”
↩︎ Reading the Spark UI: from job to task“Use the Jobs Timeline to identify major issues”
↩︎ Checkpoint - 3.
“Try again later, as most cloud provider launch failures are transient.”
↩︎ Compute event logs and termination error codes“This indicates a problem on the Databricks or cloud-provider side rather than a problem with your workload.”
↩︎ Exam trap 3“Use the error code from your cluster event log to find the relevant cause and recommended fix.”
↩︎ Checkpoint - 4.
“After approximately 20-30 minutes, check your compute event log.”
↩︎ Compute event logs and termination error codes