What you will be able to do
- Open the Spark UI and follow the documented order for diagnosing a slow job
- Spot spill and skew on a stage page, and test whether an error is really a memory problem
- Use the Streaming tab to see whether a stream is keeping up with its batch interval
- Take a thread dump of a hung task or driver, and know which access modes give you driver and executor logs
1.Opening the Spark UI and where to look first
For a Spark application, the documentation names three debugging tools: the Spark UI, driver logs and executor logs. To open the Spark UI, select your compute on the Compute page and click the Spark UI tab. It shows what your application is doing from the moment a job starts.
The Spark UI has many pages. Rather than describe each one, Databricks' diagnostic guide tells you what to look for and in what order. Start with the Jobs Timeline, which shows the major problems. Then narrow down to the longest stage and the possible reasons it is slow.
Checkpoint 1 of 6· Put it in order
Put the steps of the Spark UI diagnostic guide in order
- 1.Look at the longest stage
- 2.Use the Jobs Timeline to identify major issues
- 3.Look for skew or spill
- 4.Determine if the longest stage is I/O bound
- 5.Look for other causes of slow stage runtime
The guide goes from the timeline to one stage. It checks that stage for skew or spill first, then for I/O, and only then for other causes.
“The guide contains the following steps in order:”Source: docs.databricks.com
2.Spill, skew and memory errors on a long stage
On a long-running stage, spill is the first thing to check. Spill happens when Spark runs low on execution memory, which shuffles, joins, sorts and aggregations all use. Spark then moves data from memory to disk, and that can be expensive. Spill statistics, when there are any, appear in the details at the top of the stage page. If there are no spill statistics, the stage did not spill.
Skew is the next thing to check. It means one task, or a few, take much longer than the rest, which leaves the cluster underused. To find it, scroll down to the stage's Summary Metrics and compare Max duration with the 75th percentile duration. If the stage shows neither spill nor skew, the next question is whether it is I/O bound.
Memory problems often look like a lost executor: ExecutorLostFailure ... Remote RPC client disassociated. But error messages like this are generic and can have other causes. To check whether memory really is the problem, double the memory per core. For example, move from a worker with 4 cores and 16 GB to one with 4 cores and 32 GB. What matters is the ratio of cores to memory. If the job now takes longer to fail, or doesn't fail at all, you are on the right track. Possible causes include too few shuffle partitions, large broadcasts, UDFs, window functions without PARTITION BY, skew and streaming state.
Checkpoint 2 of 6· Check yourself
A job fails with ExecutorLostFailure and the message 'Remote RPC client disassociated'. What does the Databricks guide suggest to confirm that memory is the cause?
The message is generic, so you test the theory by changing memory per core. More workers of the same type keep the ratio unchanged.
“It's the ratio of cores to memory that matters here.”Source: docs.databricks.com
Checkpoint 3 of 6· Exam question
A Lakeflow Spark Declarative Pipeline has a streaming table with three row-level expectations declared. After a run, the table has fewer rows than expected, and the team wants to know, per expectation, how many records passed and how many were dropped for that specific run. Which single source answers this without cross-referencing any other log or metric store?
Correct answer: A — The `data_quality` field nested inside the `flow_progress` event's `details`, which lists each expectation by name alongside its `passed_records` and `failed_records` counts.
- A. The `flow_progress` event's `details` struct includes a `data_quality` sub-object with an `expectations` array; each entry names the expectation and reports `passed_records` and `failed_records` for that flow's execution, which is exactly the per-expectation breakdown asked for.
- B. Driver stderr contains general JVM and application logging, not a structured per-expectation pass/fail breakdown; expectation results are recorded as structured event log data, not printed as log lines.
- C. The Structured Streaming tab visualizes generic streaming query metrics such as input and processing rate; it has no concept of declared expectations and does not render per-expectation accept/reject counts.
- D. `runtime_details` events capture environment information such as the Databricks Runtime version and cluster configuration in use, not expectation predicates or the row counts they accepted or rejected.
Sources3
3.Checking a streaming job on the Streaming tab
The Spark UI shows a Streaming tab only while a streaming job is running on the compute. The page shows whether input events are arriving from the source. Its most important graph is Processing Time. As a rule of thumb, each batch should finish within 80% of the batch interval. If the average processing time gets close to the interval or exceeds it, batches start to queue up, a backlog builds, and the backlog can eventually bring the job down.
The page also lists the last 1,000 completed batches. Click a batch to open its details. Input shows what that batch read, such as Kafka topic, partition and offsets, or file names. Processing links to the job ID. The job page has a DAG visualization, where grayed boxes are skipped stages. Spark skips stages whose data is checkpointed or cached, so these are expected and not a fault.
Checkpoint 4 of 6· Check yourself
A streaming job should be running, but no Streaming tab appears in the Spark UI. Where should you look next?
The Streaming tab appears only when a streaming job is running. If it is missing, the stream probably failed to start, and the driver logs hold the exception.
“the Streaming job was not started because of some exception. You can drill into the Driver logs to look at the stack trace”Source: docs.databricks.com
Sources1
4.Thread dumps for hanging tasks and drivers
A thread dump is a snapshot of the state of a JVM's threads. It is the tool to use when one task hangs or runs slowly. Go from Jobs to the job, then to the stage, then to its Tasks list, and note the Task ID and Executor ID. Next open the Executors tab, find that executor's row and click Thread Dump. Then pick the thread whose name contains TID followed by your Task ID. If the task has already finished, there will be no matching thread.
The driver can hang too. You might see no Spark progress bars at all, or bars stuck at 100%. In that case, open the Executors tab and click Thread Dump in the driver row.
Checkpoint 5 of 6· Check yourself
A notebook cell's Spark progress bars have been stuck at 100% for a long time. What does the documentation suggest you look at?
Progress bars stuck at 100% are given as a sign of a driver that isn't making progress. The driver's thread dump shows what its threads are doing.
“In the Executors table, in the driver row, click the link in the Thread Dump column.”Source: docs.databricks.com
Sources1
5.Driver and executor logs, and who can read them
Driver logs contain exceptions and also the output of any print statements in the DAG. They help with a stream that never started, and with a stream whose batches never reach Completed batches. Executor logs help when particular tasks misbehave. To get to them, take the executor from the task details page, then go to the compute UI, click the number of nodes, then the master, then the worker, and open its log4j output. Whether you can see either log depends on the compute's access mode:
| Log | Standard access mode | Dedicated access mode |
|---|---|---|
| Driver logs | Workspace admins only | The dedicated user or group, and workspace admins |
| Executor logs | Not available | The dedicated user or group, and workspace admins |
Checkpoint 6 of 6· Exam question
A batch job that joins two large tables is taking far longer than expected. In the Spark UI, the engineer opens the Stages tab for the join's shuffle stage and sees a task duration summary where the maximum task duration is over ten times the median, while most tasks finish quickly. Which combination of follow-up checks in the Spark UI is most appropriate for confirming and diagnosing this pattern?
Correct answer: A — Sort the stage's task list by shuffle read size to confirm a small number of tasks handled disproportionately large partitions, and check the Executors tab for whether the slow tasks concentrated on one or two executors.
- A. A small number of tasks running far longer than the median inside one stage is the classic signature of data skew; sorting by shuffle read size confirms whether those tasks are processing disproportionately large partitions, and checking executor placement rules out an infrastructure-only cause.
- B. A blanket mismatch between `spark.sql.shuffle.partitions` and core count would make every partition undersized or oversized roughly equally, producing uniformly slow or uniformly fast tasks, not the lopsided pattern of a few very slow tasks among many fast ones described here.
- C. The stage in question is a shuffle join stage, not a stage reading cached RDD partitions, and the Storage tab reports on cached data footprint rather than per-task shuffle read skew, so it does not explain this specific symptom.
- D. Fair-scheduler pool weights govern how compute is shared across concurrently submitted jobs, not how records are distributed across tasks within a single stage, so pool weighting would not produce a skewed per-task duration pattern inside one stage.
Sources1
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A workspace admin can always open executor logs, whatever the compute's access mode.Why is that wrong?
Executor logs are not available at all on Standard access mode compute. On Dedicated access mode, the dedicated user or group and workspace admins can read them.
2.An ExecutorLostFailure message proves the job ran out of memory.Why is that wrong?
These messages are generic. Confirm a memory problem by raising memory per core and seeing whether the failure changes.
3.Grayed boxes in a streaming job's DAG mean those stages failed.Why is that wrong?
Grayed boxes are stages Spark skipped because their data was checkpointed or cached.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“select your compute from the Compute page, then click the Spark UI tab”
↩︎ Opening the Spark UI and where to look first“it is good if you can process each batch within 80% of your batch processing time.”
↩︎ Checking a streaming job on the Streaming tab“Thread dumps are useful in debugging a specific hanging or slow-running task.”
↩︎ Thread dumps for hanging tasks and drivers“If the task has finished running, you will not find a matching thread”
↩︎ Thread dumps for hanging tasks and drivers“For compute with Standard access mode, only workspace admins can access driver logs.”
↩︎ Driver and executor logs, and who can read them“Executor logs are not available for compute with Standard access mode.”
↩︎ Exam trap 1“Grayed boxes represent skipped stages.”
↩︎ Exam trap 3“the Streaming job was not started because of some exception. You can drill into the Driver logs to look at the stack trace”
↩︎ Checkpoint“In the Executors table, in the driver row, click the link in the Thread Dump column.”
↩︎ Checkpoint - 2.
“Use the Jobs Timeline to identify major issues”
↩︎ Opening the Spark UI and where to look first“The guide contains the following steps in order:”
↩︎ Checkpoint - 3.
“Spill is what happens when Spark runs low on execution memory.”
↩︎ Spill, skew and memory errors on a long stage“If you don't see any stats for spill, that means the stage doesn't have any spill.”
↩︎ Spill, skew and memory errors on a long stage“If the Max duration is 50% more than the 75th percentile, you may be suffering from skew.”
↩︎ Prediction
Also cited
“These error messages, however, are often generic and can be caused by other issues.”
↩︎ Exam trap 2“It's the ratio of cores to memory that matters here.”
↩︎ Checkpoint