What you will be able to do
- Explain when cluster-installed libraries become visible and when an uninstall takes effect
- Resolve version conflicts between notebook-scoped %pip installs and Databricks Runtime or cluster libraries
- Confirm or rule out an executor memory problem using the event log, the Spark UI and a memory-per-core test
- Choose the documented fixes for an overloaded driver
1.Compute-scoped libraries: install, update and uninstall
Compute-scoped libraries are available to every notebook attached to the cluster, and the cluster's Libraries tab shows each library's name, version, type and install status. That status column is where you see failures. A library that takes more than two hours to install is marked as failed, and a Diagnose error button appears next to the failed package. The button opens Genie Code, which diagnoses the error and suggests possible fixes. If the cluster was created from a policy that enforces library installation, you can't add or remove libraries yourself. Workspace admins manage those libraries through the policy.
Install and uninstall behave differently. An install is picked up by the next session. An uninstall waits for a restart: until then the library shows as Uninstall pending restart and is still loaded. Because of this, a version conflict can't be fixed by installing a second version next to the first. To update a cluster-installed library, uninstall the old version and install the new one. The exception is a requirements.txt file: if you change its contents, you can reinstall it without uninstalling or restarting.
Conflicts can also arise between cluster libraries themselves. For Maven libraries, Maven uses the closest-to-root version, so when two packages need different versions of a dependency the order matters, and the install may fail when the package with the older dependency loads first. The documented workaround is the Exclusions field, where you give the groupId and artifactId of the conflicting dependency, for example log4j:log4j. The installation order of Maven packages can also change the final dependency tree. For PyPI libraries used by jobs, pin the version. If it isn't fully specified, Databricks uses the latest matching version, so different runs of the same job can get different versions as new ones are published.
Checkpoint 1 of 7· Put it in order
Put the steps for removing a cluster-installed library in order
- 1.Click the Libraries tab
- 2.Click Restart and Confirm
- 3.Click the cluster name
- 4.Click Compute in the sidebar
- 5.Select the library's checkbox, click Uninstall, then Confirm
Uninstall only marks the library as Uninstall pending restart. The restart is what actually removes it, so restarting is the last step.
“When you uninstall a library from a cluster, the library is removed only when you restart the cluster.”Source: docs.databricks.com
2.Notebook-scoped %pip installs and version conflicts
Notebook-scoped libraries, installed with %pip, apply only to the current notebook and any jobs associated with it. Other notebooks on the same cluster don't see them. That makes %pip the safer way to try a different version of a package without changing the environment for everyone. Notebook-scoped libraries don't persist across sessions, though, so you must reinstall them at the start of each session or after the notebook is detached.
Most version conflicts show up as one of three problems. First, %pip doesn't restart the Python process, so after an upgrade the old version can still be loaded until you call dbutils.library.restartPython(). Second, upgrading, changing or removing core Python packages such as IPython with %pip can break features. The documented reset is to restart the cluster or start a new session. Third, the version bundled with Databricks Runtime, or one installed as a cluster library, can't be removed with %pip. If you installed a different version on top, %pip uninstall reverts to that underlying version rather than removing the package.
Checkpoint 2 of 7· Fill the gap
Which flag completes this notebook-scoped uninstall? The docs say it is required.
%pip uninstall ? matplotlibThe docs state that the -y option is required for %pip uninstall in a notebook.
Source: docs.databricks.comCheckpoint 3 of 7· Check yourself
A notebook ran %pip install matplotlib==3.8.4 over the version that ships with Databricks Runtime. What happens when it runs %pip uninstall -y matplotlib?
%pip can remove only the version you layered on top. The runtime-bundled or cluster-installed version stays, so the uninstall reverts to it.
“you cannot use a %pip command to uninstall the version of a library included in Databricks Runtime or installed on the cluster.”Source: docs.databricks.com
Checkpoint 4 of 7· Exam question
A data engineer installs a workspace library that pulls in a specific version of `ipython` as a transitive dependency. Immediately afterward, every command run in existing notebooks attached to that cluster returns `Cancelled` after roughly 30 seconds, even for trivial operations like printing a variable. What is the most likely cause?
Correct answer: A — The newly installed dependency overwrote a core package such as `ipython` or `numpy` on the Python path, breaking the REPL that notebooks use to execute commands.
- A. Installing a conflicting version of a core REPL dependency like `ipython`, `numpy`, or `pandas` onto the Python path can break the Python read-eval-print loop that notebooks depend on, and a broken REPL manifests as every subsequent command silently returning `Cancelled` after the same fixed timeout. This exactly matches the described symptom of trivial commands failing right after the install.
- B. A trivial command like printing a variable does not require executor compute, so losing executors would not explain every command, including ones with no distributed work, failing the same way. The failure pattern is specific to REPL health, not to executor availability.
- C. Installing a library does not change a notebook's default language setting, and a language mismatch would produce a syntax or interpreter error rather than a generic 30-second cancellation. This does not match the observed behavior.
- D. A single library install is very unlikely to exhaust cluster-wide DBFS storage, and running out of storage typically surfaces as a write or disk-space error rather than a uniform command cancellation after a fixed delay. The timing and error pattern point to a broken REPL, not storage exhaustion.
Sources4
3.Confirming an executor out-of-memory problem
Executor memory problems usually show up as failed jobs and removed executors. Losing executors isn't always a sign of memory pressure, though. The documented causes are autoscaling (expected, not an error), spot instance losses (the cloud provider taking back VMs), and executors running out of memory. Rule out the first two before blaming memory.
| Cause | What to conclude |
|---|---|
| Autoscaling | Expected behaviour; not an error |
| Spot instance losses | The cloud provider is reclaiming your VMs; see the compute event log |
| Executors running out of memory | Investigate as a Spark memory issue |
The diagnostic path works from the outside in. In the Spark UI, open the failing job and scroll to the failed stage and its failure reason, which is often generic. Then check the compute's event log for resizing or spot-loss events. If the event log explains nothing, open the Spark UI's Executors tab and read the failed executors' logs. If none of these turns up another cause, the likeliest explanation is memory.
Even then, a message like ExecutorLostFailure … Remote RPC client disassociated can have other causes. To confirm a memory problem, double the memory per core. For example, move from a worker with 4 cores and 16 GB to one with 4 cores and 32 GB. What matters is the ratio of memory to cores, not the total. If the job fails later or doesn't fail at all, memory was the problem. Adding memory may be the whole fix. If it isn't, or the cost is too high, the documented root causes to investigate are: too few shuffle partitions, a large broadcast, UDFs, a window function without PARTITION BY, skew, and streaming state.
Checkpoint 5 of 7· Check yourself
Your workers have 8 cores and 32 GB, and tasks fail with a generic ExecutorLostFailure. Which change is the documented test of whether this is a memory problem?
The test doubles memory per core. 16 cores with 64 GB keeps the same ratio, and adding workers doesn't change per-core memory at all.
“It's the ratio of cores to memory that matters here.”Source: docs.databricks.com
4.When the driver is the bottleneck
An overloaded driver is the usual driver-side problem, and the guidance for it is about driver size and concurrency. The most common cause is too much running on one cluster at the same time: many streams, many queries, or many Spark jobs launched from threads. Non-Spark code also keeps the driver busy while the workers sit idle. Notebook-scoped library installs add to this, because they can increase traffic to the driver node. The docs point to their driver node size recommendations for sizing.
There are three documented options: increase the size of the driver, reduce the concurrency, or spread the load across multiple clusters. Databricks recommends trying the first one first, by doubling the driver size and seeing how that affects the job.
Driver out-of-memory has its own documented causes and checks. Calling collect() or other expensive DataFrame operations on the driver can materialize large result sets and cause OOM errors, and in SparkR and sparklyr you should collect only small amounts of data. When the driver crashes from OOM or contention, monitor the driver metrics in the Spark UI. If CPU, memory or disk usage is high, adjust the driver size in the cluster compute settings. If problems persist, check that your code isn't running memory-intensive operations or UDFs on the driver. If you can't scale the driver up any further, split the jobs across multiple clusters. For stateful streaming queries, the RocksDB state store helps avoid OOM issues and GC pauses.
A reasonable answer: double the driver size first, as Databricks recommends. Then watch the driver's CPU, memory and disk usage in the Spark UI driver metrics. If usage stays high and the driver can't grow further, reduce concurrency or move some of the streams onto another cluster.
Checkpoint 6 of 7· Check yourself
A driver is overloaded because one cluster runs many concurrent streaming queries. What does Databricks recommend trying first?
Of the three documented options for an overloaded driver, Databricks recommends starting with a larger driver. Changing worker memory doesn't help the driver.
“Databricks recommends you first try doubling the size of the driver and see how that impacts your job.”Source: docs.databricks.com
5.Cluster startup failures from init scripts, libraries and capacity
When a cluster won't start, begin with the cluster event log and the error code it reports. For library and init-script problems, the cluster's init script events show which scripts started and which finished. If cluster log delivery is configured, the stdout and stderr logs of each init script are written under the cluster's init_scripts folder. Otherwise they are in /databricks/init_scripts on the node.
A SPARK_STARTUP_FAILURE means the Spark driver didn't start within the timeout, typically 200 seconds. Check Spark configuration for misconfigurations and check init scripts for errors that could delay or prevent driver startup. To isolate the cause, remove the custom Spark configs and init scripts. Hardware slowness on smaller instances can also cause this timeout, so try a different instance type. A cluster that connects to an external Hive metastore can fail to launch when it downloads all the metastore libraries from a Maven repo. Init scripts can fail the setup too, because global and cluster-named init scripts block on RPCs, so use a cluster-scoped init script instead.
For libraries, a timeout while installing is usually an intermittent network problem, and re-running the job or restarting the cluster normally fixes it. A library that can't install within two hours is marked failed. Compute that uses the Environments dependency mode (Beta) caches base environments, so dependencies don't resolve at startup. That avoids failures from package resolution, repository availability or network reliability. In that mode, don't install PySpark or any library that pulls it in as a dependency. Doing so stops the session with an error, and the fix is to remove the library and reset the environment.
Checkpoint 7 of 7· Exam question
An engineering team notices that cluster launches for a specific instance type fail sporadically throughout the day, and the cluster event log surfaces an error originating from the cloud provider about insufficient capacity in the selected availability zone. The Databricks cluster configuration itself has not changed. What should the team conclude about this failure?
Correct answer: A — The cloud provider is temporarily unable to allocate the requested instance type in that region or zone, a capacity constraint outside Databricks' control.
- A. A capacity error returned directly from the cloud provider, especially one that appears intermittently rather than consistently, is a classic sign that the provider cannot fulfill instance requests for that type or zone at that moment. Since the Databricks configuration has not changed, the constraint sits outside Databricks and typically resolves by retrying, choosing another zone, or requesting more quota.
- B. A missing IAM permission produces an authorization error and would fail every launch attempt consistently, not intermittently throughout the day. The described error also originates from the cloud provider's capacity system rather than from an access-control check.
- C. API rate limiting affects calls to the Databricks control plane, such as repeated cluster-start requests in a short window, and produces a throttling error rather than a cloud-provider capacity message. The scenario's error text specifically points to instance availability, not request throttling.
- D. A deprecated runtime is flagged during cluster configuration validation, not as a cloud-provider capacity error at launch time. Runtime deprecation would also cause consistent failures rather than sporadic ones tied to a specific instance type.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Running %pip uninstall in a notebook removes a package that ships with Databricks Runtime.Why is that wrong?
%pip removes only a version you installed on top. The runtime-bundled or cluster-installed version stays, and the notebook reverts to it.
Covered in Notebook-scoped %pip installs and version conflicts
2.An ExecutorLostFailure / 'Remote RPC client disassociated' message proves the executors ran out of memory.Why is that wrong?
These messages are generic. Rule out autoscaling and spot losses in the event log, then confirm memory by doubling memory per core.
Covered in Confirming an executor out-of-memory problem
3.Clicking Uninstall on a cluster library removes it right away, so the conflicting version is gone.Why is that wrong?
The library stays loaded, with status Uninstall pending restart, until the cluster is restarted.
Covered in Compute-scoped libraries: install, update and uninstall
Practise it for real
Pin a different version of a package in one notebook, load it, then revert to the runtime's version, without affecting other notebooks on the cluster.
1.In a notebook on classic compute, run %pip install matplotlib==3.8.4
Why: A notebook-scoped install changes only this notebook's environment, so other users on the cluster aren't affected.
You should see: The install completes. Other notebooks attached to the same cluster still see their original matplotlib version.
2.Run dbutils.library.restartPython() and then import matplotlib and print its version
Why: %pip doesn't restart the Python process, so the version that was already loaded may stay in use until you restart.
You should see: After the restart, the import reports 3.8.4.
3.Run %pip uninstall -y matplotlib, restart Python again, and check the version
Why: %pip can't remove the version bundled with Databricks Runtime or installed on the cluster. It only removes your override.
You should see: matplotlib is still importable, now at the Databricks Runtime (or cluster-installed) version.
Stuck? Get a nudge
If the -y flag is left out, the uninstall won't run: the docs say it is required.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“A library that has taken more than 2 hours to install will be marked as failed.”
↩︎ Compute-scoped libraries: install, update and uninstall“To update a cluster-installed library, uninstall the old version of the library and install a new version.”
↩︎ Compute-scoped libraries: install, update and uninstall“Until you restart the cluster, the status of the uninstalled library appears as Uninstall pending restart.”
↩︎ Exam trap 3“You must start a new session for the notebook to see the new library.”
↩︎ Prediction“When you uninstall a library from a cluster, the library is removed only when you restart the cluster.”
↩︎ Checkpoint - 2.
“Click Diagnose error to use Genie Code to help you debug.”
↩︎ Compute-scoped libraries: install, update and uninstall - 3.
“To work around this, use the Exclusions field to exclude the conflicting library.”
↩︎ Compute-scoped libraries: install, update and uninstall“The installation order of Maven packages may affect the final dependency tree”
↩︎ Compute-scoped libraries: install, update and uninstall“Specifying the library version prevents new, breaking changes in libraries from breaking your jobs.”
↩︎ Compute-scoped libraries: install, update and uninstall - 4.
“Other notebooks attached to the same cluster are not affected.”
↩︎ Notebook-scoped %pip installs and version conflicts“%pip commands do not automatically restart the Python process.”
↩︎ Notebook-scoped %pip installs and version conflicts“If you experience such problems, reset the environment by restarting the cluster or starting a new session.”
↩︎ Notebook-scoped %pip installs and version conflicts“You might experience more traffic to the driver node when working with notebook-scoped library installs.”
↩︎ When the driver is the bottleneck“you cannot use a %pip command to uninstall the version of a library included in Databricks Runtime or installed on the cluster.”
↩︎ Exam trap 1“you cannot use a %pip command to uninstall the version of a library included in Databricks Runtime or installed on the cluster.”
↩︎ Checkpoint - 5.
“you'll first want to check the compute's Event log to see if there's any explanation for why the executors failed.”
↩︎ Confirming an executor out-of-memory problem“If you've gotten this far, the likeliest explanation is a memory issue.”
↩︎ Confirming an executor out-of-memory problem - 6.
“you can verify the issue by doubling the memory per core to see if it impacts your problem.”
↩︎ Confirming an executor out-of-memory problem“that's a good sign that you're on the right track.”
↩︎ Confirming an executor out-of-memory problem“These error messages, however, are often generic and can be caused by other issues.”
↩︎ Exam trap 2“It's the ratio of cores to memory that matters here.”
↩︎ Checkpoint - 7.https://docs.databricks.com/aws/en/optimizations/spark-ui-guide/spark-driver-overloadedOfficial docs
“The most common reason for this is that there are too many concurrent things running on the cluster.”
↩︎ When the driver is the bottleneck“Databricks recommends you first try doubling the size of the driver and see how that impacts your job.”
↩︎ Checkpoint - 8.
“Monitor driver metrics in the Spark UI. If you see high CPU, memory, or disk usage, adjust driver sizing in the cluster compute settings.”
↩︎ When the driver is the bottleneck“Avoid calling collect() or other expensive DataFrame operations on the driver unless absolutely necessary”
↩︎ When the driver is the bottleneck“Use RocksDB as the state store provider to avoid OOM issues and GC pauses.”
↩︎ When the driver is the bottleneck - 9.
“Only collect small amounts of data back to R data frames, or the Spark driver will run out of memory.”
↩︎ When the driver is the bottleneck - 10.
“Check your init scripts for potential errors that could delay or prevent driver startup.”
↩︎ Cluster startup failures from init scripts, libraries and capacity“Remove custom Spark configs and init scripts to isolate the issue.”
↩︎ Cluster startup failures from init scripts, libraries and capacity“Try launching in a different availability zone, use the auto availability zone setting”
↩︎ Cluster startup failures from init scripts, libraries and capacity - 11.https://docs.databricks.com/aws/en/init-scripts/logsOfficial docs
“Cluster event logs capture two init script events: INIT_SCRIPTS_STARTED and INIT_SCRIPTS_FINISHED”
↩︎ Cluster startup failures from init scripts, libraries and capacity - 12.https://kb.databricks.com/clusters/cluster-failed-launchSecondary source
“Use a cluster-scoped init script instead of global or cluster-named init scripts.”
↩︎ Cluster startup failures from init scripts, libraries and capacity“The cluster can fail to launch if it has a connection to an external Hive metastore”
↩︎ Cluster startup failures from init scripts, libraries and capacity“Usually you can fix this problem by re-running the job or restarting the cluster.”
↩︎ Cluster startup failures from init scripts, libraries and capacity - 13.
“Doing so will stop your session and result in an error.”
↩︎ Cluster startup failures from init scripts, libraries and capacity