What you will be able to do
- Explain who deploys and manages classic compute and who manages serverless compute
- Tell all-purpose, jobs and pipeline compute apart, including access modes and how policies limit compute creation
- Recognise the serverless limitations that force a workload back onto classic compute
- Describe how compute choice affects cost: idle time, termination behaviour and billing in system.billing.usage
- Select the most suitable compute type for a given workload: job task, SQL task, pipeline, interactive work or basic exploration
Key concept
Classic vs serverless compute — Classic compute runs in your own cloud account, and you create, size and manage it. Serverless compute runs in a compute plane that Databricks manages, so Databricks allocates, scales and upgrades it for you. Most trade-offs in latency, control, limitations and cost come from this split.
1.Who runs the machines: classic vs serverless
When you pick compute on Databricks, the first question is whose cloud account the machines run in and who manages them. Classic compute covers all-purpose, jobs and Lakeflow pipelines compute. It is deployed in your own cloud provider account, and you create, configure and manage it. Serverless compute runs in the serverless compute plane, which Databricks manages. You run workloads without provisioning anything in your cloud account, and Databricks allocates the resources for you.
The rest of this lesson follows from that difference. Databricks owns the serverless capacity, so it can hand out and scale resources quickly. The documentation says this speeds up start-up and scaling times, minimizes idle time, and reduces the need to manage compute. You own classic compute, so you keep the settings that serverless removes: custom cluster settings, compute policies, init scripts, instance pools and the Spark UI. Exam questions on compute usually ask you to weigh control against convenience and cost.
One scoping note. In the documentation, "serverless compute" means serverless for notebooks, workflows and Lakeflow pipelines. Serverless SQL warehouses also run on serverless infrastructure, but you configure them separately. They get their own treatment when you study SQL warehouses.
Checkpoint 1 of 7· Check yourself
Which statement correctly describes where classic compute runs and who manages it?
Classic compute (all-purpose, jobs and pipelines) is deployed in your cloud provider account and you manage it. Serverless is the type that Databricks deploys and manages.
“Classic compute resources are deployed in your cloud provider account, and are not managed by Databricks.”Source: docs.databricks.com
No. The documentation says serverless SQL warehouses, model serving and AI features use serverless infrastructure independently and have their own configuration paths. The serverless compute page covers only notebooks, workflows and Lakeflow pipelines.
2.Classic compute: all-purpose, jobs and pipeline compute
Classic compute comes in three kinds: all-purpose compute for interactive work, jobs compute for automated job tasks, and pipeline compute for Lakeflow pipelines. You size and configure each one yourself.
All-purpose and jobs compute have an access mode. It decides who can attach to the resource:
- Standard lets any user with CAN ATTACH TO permission attach and run workloads on the same compute at the same time. Lakeguard isolates each user's code, which allows cost savings through sharing. - Dedicated assigns the compute to a single user or group. Only that user or group can attach to it.
Entitlements decide who can create compute. Workspace admins can create any type of compute. Non-admin users with the Unrestricted cluster creation entitlement can use every configuration setting. Everyone else can only use compute they have been granted, or create compute from a compute policy they have permission on. Policies have minimal configuration options and are designed to be efficient with their default settings. Permissions on a single compute resource come in three levels: CAN ATTACH TO (attach a notebook, view metrics and the Spark UI), CAN RESTART (start, restart, terminate), and CAN MANAGE (edit details, permissions and size).
Checkpoint 2 of 7· Check yourself
A non-admin user does not have the Unrestricted cluster creation entitlement. Which statement describes the compute they can use?
Without the entitlement, access is limited to granted compute or compute created from a policy the user has permission on. Policies have minimal configuration options and efficient defaults.
“Non-admin users without the Unrestricted cluster creation entitlement can only access compute they are granted permissions to or compute they create using policies”Source: docs.databricks.com
Checkpoint 3 of 7· Check yourself
Five analysts want to share one classic all-purpose compute resource to save cost, and each runs their own code concurrently with their code isolated from the others. Which access mode fits?
Standard access mode lets any user with CAN ATTACH TO run workloads concurrently on the same compute, with Lakeguard isolating each user's code. Dedicated compute is assigned to a single user or group, so it is not the concurrent sharing mode.
“Allows any user with CAN ATTACH TO permission to attach to and concurrently execute workloads on the compute resource”Source: docs.databricks.com
Matching classic compute to the workload. Each kind of classic compute has a job it is meant for:
- Classic jobs compute is for non-SQL jobs that need custom cluster settings that serverless does not offer. For most other automated work, Databricks recommends serverless instead. - Classic pipeline compute is for pipelines that need a feature serverless does not support, or that work with the legacy Hive metastore. - Classic all-purpose compute is for interactive work, and for workloads that need RDD APIs or R language support. Generally avoid it for automated jobs. It is billed at a different rate than jobs compute, it is often shared across teams, so scheduled jobs on it can see more latency, and it stops on inactivity rather than at the end of a run.
There are two documented reasons to run a job on all-purpose compute anyway. One is iterative development or testing of a new job, because jobs compute start-up times make iteration tedious. The other is a short-lived job that runs often or on a fixed schedule, because a running all-purpose compute has no start-up time. Watch the idle-time cost in that second case.
3.Serverless compute: characteristics and limitations
Serverless compute is available by default in most workspaces and needs no enablement. It does have one hard requirement: Unity Catalog. Legacy workspaces without Unity Catalog cannot use serverless compute until they upgrade. Serverless is also versionless. Databricks upgrades the runtime automatically, so workloads always run on the latest runtime version.
The convenience has limits. Serverless uses Spark Connect, which shapes most of the limitations below.
| Area | On serverless compute | Documented alternative |
|---|---|---|
| Languages and APIs | R is not supported; only Spark Connect APIs, no RDD APIs | Classic all-purpose compute for RDD or R |
| Run duration | Maximum runtime of 7 days; longer runs are terminated and not retried | Break into smaller runs or use classic compute |
| Compute configuration | No compute policies, compute-scoped init scripts, compute-scoped libraries, instance pools or compute event logs; most Apache Spark compute configurations are not supported | Base environment or notebook-scoped libraries |
| Streaming triggers | Trigger.AvailableNow() supported and recommended; Trigger.Once() supported but deprecated; Trigger.Continuous and Trigger.ProcessingTime not supported | Trigger.AvailableNow() in a continuous job, or a continuous pipeline |
| Caching | df.cache(), df.persist() and CACHE TABLE raise an exception | None listed |
| Debugging | Spark UI and Spark logs not available | Query profile |
| Data access | Unity Catalog required for external sources; DBFS access limited | External locations, Unity Catalog volumes or workspace files |
Checkpoint 4 of 7· Check yourself
A backfill job is expected to run for ten days without stopping. What does the documentation recommend?
Serverless runs are capped at 7 days. Runs that exceed the cap are terminated and not retried, so you either split the work or move it to classic compute.
“To run workloads longer than 7 days, break them into smaller runs or use classic compute.”Source: docs.databricks.com
4.How compute is billed, and what idle time costs
The sources describe the cost model by how it works, not with price lists. They give no per-DBU rates, so this lesson does not quote any. For pricing per SQL warehouse type, the documentation points to the Databricks SQL pricing page. Four facts carry the exam weight.
Pricing follows the compute. The cost of a Lakeflow Job depends on the compute its tasks run on. Total cost is made of Databricks Units (DBUs) plus virtual machine, disk and associated network costs. Databricks bills all-purpose compute at a different rate than jobs compute, which is one reason it recommends against all-purpose compute for jobs. Databricks describes serverless as generally recommended because it offers faster start-up, automatic scaling and lower cost. Much of that saving comes from minimizing idle time. For jobs and pipelines on serverless you also pick performance optimized or standard performance mode based on your latency and cost needs.
Classic compute charges you for idle time, twice over. Idle classic compute keeps accumulating both DBU charges and cloud instance charges during the inactivity period before it terminates. Serverless differs: for serverless services the DBU cost already includes the virtual machine costs, and serverless minimizes idle time. A serverless SQL warehouse also downscales quickly when demand is low to minimize costs. Pro and classic SQL warehouses run their compute layer in your AWS account, so they carry your own cloud costs too.
Termination is how you stop paying for idle classic compute. You can terminate classic compute manually, or configure it to terminate automatically after a set period of inactivity. The inactivity period must be between 10 and 10,000 minutes (about 7 days). You opt out by clearing the Auto Termination checkbox or by specifying 0. Compute counts as inactive when all commands, including Spark jobs, Structured Streaming and JDBC calls, have finished. A terminated compute keeps its configuration for reuse, but unless it is pinned or restarted it is permanently deleted 30 days after termination. Jobs compute avoids idle cost by design: a job run on new job compute terminates the compute when the job completes, and that compute cannot be restarted. In contrast, a job scheduled on an existing all-purpose compute that has been terminated will autostart it. If several tasks share one job compute, they start up faster. The trade-off is that the compute stays on, idle, while tasks that do not use it are running. A shared job cluster is scoped to a single job run.
Serverless costs are tracked in system tables. To find what a serverless workload cost, query system.billing.usage. Its usage_metadata column identifies the job or notebook, and identity_metadata.run_as shows whose credentials ran it. One workload can produce several records. Their DBUs add up to the hourly consumption for that run. Usage can take up to 24 hours to appear. Some background features, including predictive optimization and data quality monitoring, are billed under the serverless jobs SKU even if you never ran a serverless notebook. To estimate serverless costs, Databricks recommends benchmarking a representative workload and then analyzing the billing table.
Checkpoint 5 of 7· Check yourself
An engineer queries system.billing.usage an hour after a serverless job finished and finds no rows for it. What is the most likely explanation?
The billing table is the documented way to see serverless cost, but records can lag the run by up to a day.
“there could be up to a 24-hour delay between when you run a workload and its usage being reflected”Source: docs.databricks.com
Model answer for the reflection. With auto-termination switched off (inactivity period 0), the cluster never terminates after the nightly job finishes. Between runs it sits idle, and idle classic compute keeps accumulating both DBU charges and cloud instance charges. They also pay the all-purpose rate, which differs from the jobs compute rate. Running the job on new jobs compute would end the compute when the run completes. Setting an inactivity period of 10 minutes or more would also stop the idle charges.
Checkpoint 6 of 7· Exam question
A BI team refreshes an executive dashboard throughout the business day and needs a SQL compute resource that resumes from idle in a few seconds and scales quickly as concurrent users open the dashboard during peak hours, with no infrastructure to size or manage. Which compute option should the team choose?
Correct answer: A — Provision a serverless SQL warehouse, since it starts in a few seconds and lets Databricks scale clusters automatically as query load changes.
- A. A serverless SQL warehouse resumes in roughly 2 to 6 seconds and its Intelligent Workload Management engine adds or removes clusters automatically as concurrent dashboard queries rise and fall. This matches the requirement for rapid resume and hands-off scaling with no capacity to size in advance.
- B. A classic SQL warehouse typically takes several minutes to start from idle and scales less responsively than serverless, which fails the requirement for a fast dashboard refresh. Cost per DBU is not the deciding factor when the core requirement is startup latency and quick concurrency scaling.
- C. A Pro SQL warehouse still takes minutes to start and is chosen for custom networking, hybrid connectivity, or regions without serverless support, none of which this dashboard scenario mentions. It does not resume as quickly as serverless compute, so it does not fit the stated need.
- D. An all-purpose cluster is built for shared interactive notebook sessions rather than fast-resuming, highly concurrent BI query serving, and it does not offer the few-second resume time serverless warehouses provide. Running SQL through a notebook cluster also adds operational overhead the team wants to avoid.
5.Choosing the right compute for each workload
Selecting compute is a matter of matching the workload to the type the documentation recommends, and moving to another type only when a specific need forces it.
Default to serverless. Databricks calls serverless compute generally recommended, and the recommended choice for most automated workloads. It has fewer settings to manage, faster start-up, automatic scaling and lower cost. For jobs and pipelines you can choose performance optimized or standard performance mode based on your latency and cost needs.
Pick by task type. For SQL tasks and dbt, use a SQL warehouse, ideally serverless. For notebooks, Python scripts and Python wheels, use serverless jobs. For Lakeflow pipelines, use a serverless pipeline. JAR and Spark Submit tasks have classic jobs as the recommended compute.
Pick a SQL warehouse type by need. A serverless SQL warehouse is recommended for BI, ETL and exploratory analysis, with start-up typically 2 to 6 seconds. A pro SQL warehouse fits when serverless is not available or you need custom networking, for example federation or a hybrid setup. A classic SQL warehouse is entry-level performance, for basic interactive exploration when serverless or pro are not options. Pro and classic warehouses take several minutes to start, typically about 4 minutes.
Fall back to classic only for a reason. Choose classic jobs compute for non-SQL jobs that need custom cluster settings serverless lacks, classic pipeline compute for features serverless does not support, and classic all-purpose compute for RDD APIs or R.
| Workload | Recommended compute | Choose something else when |
|---|---|---|
| Notebook, Python script or Python wheel task in a job | Serverless jobs | Non-SQL job needs custom cluster settings not available in serverless: classic jobs |
| SQL task or dbt | Serverless SQL warehouse | Serverless is not available or you need custom networking: pro SQL warehouse |
| Lakeflow pipelines | Serverless pipeline | A feature is not supported in serverless, or you use the legacy Hive metastore: classic pipeline |
| JAR or Spark Submit | Classic jobs | Not applicable |
| Interactive work that needs RDD APIs or R | Classic all-purpose compute | Not applicable |
| Basic interactive exploration | Serverless SQL warehouse | Serverless and pro are not options: classic SQL warehouse |
| Iteratively developing or testing a new job | Serverless compute for jobs, the recommended substitute for most task types | You might choose classic all-purpose compute because jobs compute start-up times make iteration tedious |
Checkpoint 7 of 7· Check yourself
A job task runs a JAR. According to the documentation's table of recommended compute per task type, which compute is recommended?
For JAR tasks the recommended compute is classic jobs. Serverless jobs are recommended for notebook, Python script and Python wheel tasks, and SQL warehouses are for SQL and dbt tasks.
“JAR | Classic jobs | Classic jobs, classic all-purpose”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Serverless compute is available in every workspace, including legacy workspaces that still use only the Hive metastore.Why is that wrong?
Serverless compute requires Unity Catalog. Legacy workspaces without it must upgrade before they can use serverless.
Covered in Serverless compute: characteristics and limitations
2.Job compute can be restarted after a job finishes, just like an all-purpose cluster.Why is that wrong?
New job compute terminates when the job completes and cannot be restarted. A terminated all-purpose compute that a job is scheduled on autostarts instead.
3.Classic all-purpose compute is the best choice for scheduled production jobs because it is always available.Why is that wrong?
The documentation says to generally avoid all-purpose compute for automated workloads, because it is not optimized for them. Serverless is recommended for most automated workloads.
Covered in Choosing the right compute for each workload
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Classic compute resources are deployed in your cloud provider account, and are not managed by Databricks.”
↩︎ Who runs the machines: classic vs serverless“Classic compute refers to all-purpose, jobs, and Lakeflow pipelines compute resources that you create, configure, and manage for your workloads.”
↩︎ Classic compute: all-purpose, jobs and pipeline compute“The compute resource is assigned to a single user or group.”
↩︎ Classic compute: all-purpose, jobs and pipeline compute“Non-admin users without the Unrestricted cluster creation entitlement can only access compute they are granted permissions to or compute they create using policies”
↩︎ Classic compute: all-purpose, jobs and pipeline compute“Classic compute is different from serverless compute, which is deployed and managed by Databricks.”
↩︎ Key concept“Allows any user with CAN ATTACH TO permission to attach to and concurrently execute workloads on the compute resource”
↩︎ Checkpoint - 2.
“This speeds up start-up and scaling times, minimizes idle time, and reduces the need to manage compute resources.”
↩︎ Who runs the machines: classic vs serverless“you can run workloads without provisioning any compute resources in your cloud account.”
↩︎ Who runs the machines: classic vs serverless“Serverless workloads always run on the latest runtime version.”
↩︎ Serverless compute: characteristics and limitations“Databricks recommends running and benchmarking a representative or specific workload and then analyzing the billing system table.”
↩︎ How compute is billed, and what idle time costs“Legacy workspaces that are not enabled for Unity Catalog do not have access to serverless compute.”
↩︎ Exam trap 1“Serverless compute resources run in the serverless compute plane, which is managed by Databricks.”
↩︎ Prediction“there could be up to a 24-hour delay between when you run a workload and its usage being reflected”
↩︎ Checkpoint - 3.
“For non-SQL jobs that require custom cluster settings not available in serverless.”
↩︎ Classic compute: all-purpose, jobs and pipeline compute“Generally recommended. Offers faster start-up, automatic scaling, and lower cost.”
↩︎ How compute is billed, and what idle time costs“Recommended choice for most automated workloads.”
↩︎ Choosing the right compute for each workload“Use when serverless is not available or you require custom networking (for example, federation or hybrid).”
↩︎ Choosing the right compute for each workload“Use for basic interactive exploration when serverless or pro are not options.”
↩︎ Choosing the right compute for each workload“Generally avoid. Not optimized for automated workloads.”
↩︎ Exam trap 3 - 4.
“You are iteratively developing or testing new jobs.”
↩︎ Classic compute: all-purpose, jobs and pipeline compute“Databricks bills for all-purpose compute at a different rate than jobs compute.”
↩︎ Classic compute: all-purpose, jobs and pipeline compute“Databricks bills for all-purpose compute at a different rate than jobs compute.”
↩︎ How compute is billed, and what idle time costs“Serverless compute for jobs is the recommended substitute for most task types you might consider running against all-purpose compute.”
↩︎ Choosing the right compute for each workload - 5.
“Only Spark Connect APIs are supported. Spark RDD APIs are not supported.”
↩︎ Serverless compute: characteristics and limitations“Dataframe and SQL cache APIs are not supported on serverless compute.”
↩︎ Serverless compute: characteristics and limitations“Trigger.AvailableNow(). Databricks recommends this trigger mode for serverless compute.”
↩︎ Serverless compute: characteristics and limitations“Trigger.Once(). This deprecated mode is supported but not recommended.”
↩︎ Serverless compute: characteristics and limitations“Most Apache Spark compute configurations.”
↩︎ Serverless compute: characteristics and limitations“JAR tasks in jobs are supported.”
↩︎ Prediction“To run workloads longer than 7 days, break them into smaller runs or use classic compute.”
↩︎ Checkpoint - 6.https://docs.databricks.com/aws/en/jobs/computeOfficial docs
“Pricing for Lakeflow Jobs is tied to the compute used to run tasks.”
↩︎ How compute is billed, and what idle time costs“Compute remains on and idle while tasks not using the compute resource run.”
↩︎ How compute is billed, and what idle time costs“Serverless compute for jobs has limitations and does not support all workloads.”
↩︎ Choosing the right compute for each workload“JAR | Classic jobs | Classic jobs, classic all-purpose”
↩︎ Checkpoint - 7.
“You can manually terminate a compute or configure the compute to terminate automatically after a specified period of inactivity.”
↩︎ How compute is billed, and what idle time costs“Idle compute continue to accumulate DBU and cloud instance charges during the inactivity period before termination.”
↩︎ How compute is billed, and what idle time costs“The inactivity period must be between 10 and 10,000 minutes (about 7 days).”
↩︎ How compute is billed, and what idle time costs“You can opt out of auto termination by clearing the Auto Termination checkbox or by specifying an inactivity period of 0.”
↩︎ How compute is billed, and what idle time costs“Unless a compute is pinned or restarted, it is automatically and permanently deleted 30 days after termination.”
↩︎ How compute is billed, and what idle time costs“the compute terminates and is unavailable for restarting when the job is complete.”
↩︎ Exam trap 2 - 8.https://docs.databricks.com/aws/en/lakehouse-architecture/cost-optimization/best-practicesOfficial docs
“Note that for serverless services, the DBU cost already includes the virtual machine costs.”
↩︎ How compute is billed, and what idle time costs - 9.
“Quick downscaling to minimize costs when demand is low, providing consistent performance with optimized costs and resources.”
↩︎ How compute is billed, and what idle time costs - 10.
“The sum of these DBUs represents the hourly DBU consumption for a given job run in that timeframe.”
↩︎ How compute is billed, and what idle time costs