What you will be able to do
- Name the three technologies the Databricks lakehouse is built on and what each contributes
- Describe how an account, its workspaces and Unity Catalog metastores relate
- Say where the control plane, the classic compute plane, the serverless compute plane and workspace storage live
- Tell all-purpose clusters from job clusters, and serverless, pro and classic SQL warehouses from each other
Key concept
Data lakehouse — A single data system that gives you the cheap, open storage of a data lake and the reliability and query performance of a data warehouse. On Databricks, every platform component (the compute engine, the table format and the governance layer) works on that one shared copy of the data.
1.The lakehouse and its three building blocks
The exam calls the product the Databricks Data Intelligence Platform. The documentation describes Databricks as a unified, open analytics platform for building and running data, analytics and AI solutions. The 'intelligence' part means the platform uses AI together with the data lakehouse to learn what your data means. It then tunes performance and manages infrastructure for you automatically. It also integrates with the storage and security in your own cloud account, and it deploys cloud infrastructure on your behalf.
The data lakehouse is the base layer. Without one, teams typically keep a data lake for cheap storage and ML work and a separate data warehouse for BI. With a lakehouse, data engineers, data scientists, analysts and production systems all use one system as their single source of truth, so there are no parallel copies to keep in sync. It works by putting an optimised metadata layer on top of validated data, stored in standard formats in cloud object storage.
Databricks is built on Apache Spark, a massively scalable engine that runs on compute kept separate from storage. On top of Spark, the Databricks lakehouse adds two more core technologies. Delta Lake is the optimised storage layer and supports ACID transactions and schema enforcement. Unity Catalog is the unified, fine-grained governance solution for data and AI. A common question shape: Spark processes the data, Delta Lake makes the tables reliable, and Unity Catalog controls who can reach them.
Checkpoint 1 of 5· Check yourself
Besides Apache Spark, which two technologies does the Databricks lakehouse name as its key building blocks?
The lakehouse page lists Delta Lake (the storage layer, which provides ACID transactions) and Unity Catalog (governance) as the two additional key technologies. MLflow and Redash are open-source projects Databricks created, but the lakehouse page does not name them as its foundation.
“Delta Lake: an optimized storage layer that supports ACID transactions and schema enforcement.”Source: docs.databricks.com
2.Account, workspaces, and metastores
The Databricks account is the top-level construct for running Databricks across an organisation. At account level you manage identity and access (users, groups, service principals, SSO), create and delete workspaces across regions, create Unity Catalog metastores and attach them to workspaces, and handle billing and policies. One account can hold many workspaces and many metastores. Databricks bills in Databricks units (DBUs): units of processing capability per hour, based on the VM instance type.
A workspace is a Databricks deployment in the cloud. It is the collaboration environment where users run compute workloads: ingestion, interactive exploration, scheduled jobs and ML training. A workspace organises objects such as notebooks, libraries, dashboards and experiments into folders, and gives access to data objects and compute. An organisation can run one workspace or several.
A Unity Catalog metastore is the central governance system for data assets such as tables and ML models. Metastores are attached to workspaces. You can link one metastore to several workspaces in the same region. Each of those workspaces then sees the same data, and you manage data access controls across all of them in one place.
The same catalogs, schemas and tables. Linking one metastore to several workspaces in a region gives each of them the same view of the data, and access controls are managed across all linked workspaces.
Checkpoint 2 of 5· Match them up
Match each construct to its role
Tap a term, then the definition that fits it.
The account manages workspaces, identity and billing. Users work inside workspaces. The metastore governs the data and can be shared by every workspace in a region.
“An account can contain multiple workspaces and Unity Catalog metastores.”Source: docs.databricks.com
3.Control plane and compute plane
Databricks runs out of two planes. The control plane holds the backend services that Databricks manages, including the web application. It lives in the Databricks account, not in your cloud account. The compute plane is where your data is actually processed, and it comes in two kinds depending on the compute you use.
With classic compute, the resources run in your AWS account, in what is called the classic compute plane. New compute resources are created inside each workspace's virtual network in your account. This gives natural isolation, because each customer's compute sits in that customer's own AWS account. With serverless compute, the resources run in a serverless compute plane inside your Databricks account. Databricks creates it in the same AWS region as the workspace's classic compute plane, and wraps it in a network boundary per workspace, with several security layers that keep different customers' workspaces apart.
| Component | Location | What it contains or does |
|---|---|---|
| Control plane | Databricks account, not your cloud account | Backend services Databricks manages, including the web application |
| Classic compute plane | Your AWS account, in the workspace's virtual network | Classic compute resources that process your data |
| Serverless compute plane | Your Databricks account, same AWS region as the workspace | Serverless compute resources, isolated by a workspace network boundary |
| Workspace storage bucket (classic workspaces) | Your AWS account | Workspace system data, the default workspace catalog, legacy DBFS |
Workspaces also need storage. A classic workspace has a workspace storage bucket in your AWS account. It holds workspace system data such as query results, job run results, notebook revisions and cluster logs. If Unity Catalog was enabled automatically, it also holds the default workspace catalog, along with the legacy DBFS root. Serverless workspaces use default storage instead, which Databricks fully manages. In both cases, workspace storage is separate from your own data objects, such as Unity Catalog tables and volumes. Never delete or modify it: a workspace depends on both its control plane databases and its workspace storage, and a workspace whose storage has been deleted cannot be recovered. Databricks has deprecated storing data in the DBFS root or DBFS mounts and recommends Unity Catalog for all data access.
Checkpoint 3 of 5· Exam question
A BI team runs ad hoc dashboard queries that spike heavily during business hours and drop to almost nothing overnight. Leadership wants queries to start almost instantly when a user opens a dashboard, but does not want to pay for idle compute overnight. Which SQL warehouse type best fits this pattern?
Correct answer: A — A serverless SQL warehouse, which starts in a few seconds and scales down rapidly when query volume drops
- A. Serverless SQL warehouses run in Databricks-managed capacity in the same region as the workspace and typically start in a few seconds, and they scale down aggressively during idle periods, matching a bursty, cost-sensitive BI workload.
- B. Classic SQL warehouses take several minutes to start and only provide the Photon engine without predictive I/O or intelligent workload management, so they cannot meet the near-instant startup requirement.
- C. Pro SQL warehouses run in the customer's own cloud account and also take several minutes to spin up, which fails the near-instant startup requirement even though they support more advanced features than classic.
- D. All-purpose clusters are built for collaborative notebook and job development, not for serving low-latency SQL dashboard queries, so they are not the right compute type here.
Checkpoint 4 of 5· Check yourself
In a classic workspace, which component lives outside the customer's AWS account?
The control plane is in the Databricks account. In a classic workspace, the compute resources, their virtual network and the workspace storage bucket are all in the customer's AWS account.
“The control plane is located in the Databricks account, not your cloud account.”Source: docs.databricks.com
Sources3
4.Compute: clusters and SQL warehouses
A cluster is a set of compute resources and configurations on which you run notebooks and jobs. There are two types. You create an all-purpose cluster yourself through the UI, CLI or REST API. You can terminate and restart it manually, and several users can share it for collaborative, interactive analysis. A job cluster is created by the Databricks job scheduler when you run a job on a new job cluster. The scheduler terminates it when the job finishes, and you cannot restart it.
| Aspect | All-purpose cluster | Job cluster |
|---|---|---|
| Created by | You, through the UI, CLI or REST API | The Databricks job scheduler, when a job runs on a new job cluster |
| Restart | Can be terminated and restarted manually | Cannot be restarted; terminated when the job completes |
| Workload type | Data analytics (interactive) | Data engineering (automated) |
Databricks sorts workloads into two types: data engineering (job) workloads run on a job cluster, and data analytics (interactive) workloads run on an all-purpose cluster. A scheduled job that runs on an existing all-purpose cluster still counts as an interactive workload.
A SQL warehouse is a compute resource for running SQL queries. There are three types (classic, pro and serverless), and Databricks recommends serverless where it is available. A serverless warehouse supports every Databricks SQL performance feature. It typically starts in 2 to 6 seconds, scales up quickly to keep latency low, and scales down quickly to cut costs when demand drops. A pro warehouse supports Photon and Predictive IO but not Intelligent Workload Management. Its compute layer sits in your AWS account, and it takes about 4 minutes to start. Use pro when serverless is not available in your region, or when you need custom networking for federation or a hybrid architecture. A classic warehouse supports only Photon, gives entry-level performance, also runs in your account, and also takes about 4 minutes to start.
| Type | Photon | Predictive IO | Intelligent Workload Management | Compute layer | Typical startup |
|---|---|---|---|---|---|
| Serverless | Yes | Yes | Yes | Your Databricks account | 2 to 6 seconds |
| Pro | Yes | Yes | No | Your AWS account | About 4 minutes |
| Classic | Yes | No | No | Your AWS account | About 4 minutes |
Checkpoint 5 of 5· Check yourself
Serverless SQL warehouses are not available in your region, but your BI team needs Photon and Predictive IO. Which warehouse type fits?
Pro supports both Photon and Predictive IO and is the documented choice when serverless isn't offered in a region. Classic supports Photon only.
“A pro SQL warehouse supports Photon and Predictive IO, but does not support Intelligent Workload Management.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.All Databricks compute runs inside the customer's own AWS account.Why is that wrong?
Only classic compute runs in your AWS account. Serverless compute runs in a serverless compute plane in your Databricks account.
Covered in Control plane and compute plane
2.After a failed run, you can restart a job cluster just like an all-purpose cluster.Why is that wrong?
The job scheduler creates a job cluster for the run and terminates it when the run ends, and it cannot be restarted. Only all-purpose clusters can be restarted manually.
Covered in Compute: clusters and SQL warehouses
3.A scheduled job counts as a data engineering workload whatever cluster it runs on.Why is that wrong?
A job that runs on an existing all-purpose cluster is treated as an interactive (data analytics) workload.
Covered in Compute: clusters and SQL warehouses
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/introductionOfficial docs
“Databricks uses AI with the data lakehouse to understand the unique semantics of your data.”
↩︎ The lakehouse and its three building blocks - 2.https://docs.databricks.com/aws/en/lakehouseOfficial docs
“Apache Spark enables a massively scalable engine that runs on compute resources decoupled from storage.”
↩︎ The lakehouse and its three building blocks“Unity Catalog: a unified, fine-grained governance solution for data and AI.”
↩︎ The lakehouse and its three building blocks“A data lakehouse is a data management system that combines the benefits of data lakes and data warehouses.”
↩︎ Key concept“Delta Lake: an optimized storage layer that supports ACID transactions and schema enforcement.”
↩︎ Checkpoint - 3.
“You can link a single metastore to multiple Databricks workspaces in the same region, giving each workspace the same data view.”
↩︎ Account, workspaces, and metastores“Databricks operates out of a control plane and a compute plane.”
↩︎ Control plane and compute plane“If workspace storage is deleted, the workspace cannot be recovered.”
↩︎ Control plane and compute plane“For classic Databricks compute, the compute resources are in your AWS account in what is called the classic compute plane.”
↩︎ Exam trap 1“An account can contain multiple workspaces and Unity Catalog metastores.”
↩︎ Checkpoint“For serverless compute, the serverless compute resources run in a serverless compute plane in your Databricks account.”
↩︎ Prediction“The control plane is located in the Databricks account, not your cloud account.”
↩︎ Checkpoint - 4.
“Databricks bills based on Databricks units (DBUs), which are units of processing capability per hour based on VM instance type.”
↩︎ Account, workspaces, and metastores“There are two types of clusters: all-purpose and job.”
↩︎ Compute: clusters and SQL warehouses“You cannot restart a job cluster.”
↩︎ Exam trap 2“However, running a job on an existing all-purpose cluster is also treated as an interactive workload.”
↩︎ Exam trap 3 - 5.
“the compute layer exists in your AWS account account rather than in your Databricks account”
↩︎ Compute: clusters and SQL warehouses“Rapid startup time (typically between 2 and 6 seconds).”
↩︎ Compute: clusters and SQL warehouses“A pro SQL warehouse supports Photon and Predictive IO, but does not support Intelligent Workload Management.”
↩︎ Checkpoint