What you will be able to do
- Explain why only Unity Catalog managed tables can hand file-lifecycle maintenance to the platform
- Describe which maintenance operations predictive optimization runs, and how it chooses when to run them
- Enable, override and verify predictive optimization at account, catalog, schema and table level
- Spot what predictive optimization will not do: ZORDER, external tables, and VACUUM retention below 7 days
Key concept
Managed file lifecycle — For a Unity Catalog managed table, Unity Catalog owns both governance and the physical data files: where they live, how they are organized and optimized, and when they are deleted. Automated maintenance depends on this. The platform can only maintain files it controls.
1.Who owns the files decides who does the maintenance
Every table registered in Unity Catalog is governed: access control, auditing and lineage apply to it. For overhead, the question that matters is who controls the data files. With an external table, you choose the storage path and you look after the files: compaction, cleaning up unreferenced files, and deleting data after a drop. With a managed table, Unity Catalog chooses the storage location (inside your own cloud account) and runs the file lifecycle itself.
That split explains everything else in this lesson. Automatic compaction, automatic VACUUM and automatic statistics collection need the platform to own the files. Databricks calls managed tables the default and recommended table type, and says they cost less to store and query, and maintain and optimize themselves automatically.
| Property | Unity Catalog managed table | External table |
|---|---|---|
| Storage location | Set by Unity Catalog (in your cloud account) | Set by you |
| File lifecycle management | Managed by Unity Catalog (optimization, organization, deletion) | Managed by you |
| Drop behavior | Data files are deleted automatically after a retention period | Data files remain in place |
| Data ownership | Yes | Yes |
Dropping a table shows the difference clearly. When you drop a managed table, Databricks deletes its data files in cloud storage once the recovery period expires, and you can configure that period at catalog or schema level. When you drop an external table, only the metadata goes, so someone still has to remove the files by hand. Managed status also makes several features available that external and foreign tables don't get, including predictive optimization and automatic liquid clustering. Managed tables don't lock you in either: external engines can still reach them through open APIs such as the Unity REST API and the Iceberg REST Catalog.
Checkpoint 1 of 6· Check yourself
A team drops an external Delta table that is registered in Unity Catalog. What happens to its data files?
Automatic file deletion after DROP TABLE only applies to managed tables. For an external table, Unity Catalog removes the metadata and leaves the files where they are.
“For external tables, you must manually delete the files from your storage bucket.”Source: docs.databricks.com
2.What predictive optimization runs for you
Without predictive optimization, keeping a table healthy is a scheduling job. Someone has to decide how often to run OPTIMIZE, when to VACUUM, and when to refresh statistics, and then investigate when a query slows down because one of those was missed. Predictive optimization does this work for Unity Catalog managed tables (Delta Lake and Iceberg). It works out which tables would benefit from maintenance, queues only those operations, and collects statistics when data is written. Tables that don't need maintenance aren't touched, so no compute is spent on unnecessary runs.
| Operation | What it does on a managed table |
|---|---|
| OPTIMIZE | Triggers incremental clustering for enabled tables and improves query performance by optimizing file sizes; never runs ZORDER |
| VACUUM | Reduces storage costs by deleting data files no longer referenced by the table |
| ANALYZE | Scans the table and collects statistics to improve query performance |
The work isn't free. Predictive optimization runs these operations on serverless compute for jobs, and they're billed under a serverless jobs SKU. What you save is engineering effort and wasted runs, not the compute itself. To see what it did and what it cost, query the system table system.storage.predictive_optimization_operations_history.
VACUUM retention is controlled by delta.deletedFileRetentionDuration, which defaults to 7 days. If you need longer time travel, set this property before you enable predictive optimization. If you set it below 7 days, predictive optimization still keeps files for at least 7 days when it runs VACUUM FULL, which prevents data loss.
ALTER TABLE table_name SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = '30 days');Checkpoint 2 of 6· Match them up
Match each predictive optimization operation to its effect
Tap a term, then the definition that fits it.
Predictive optimization runs exactly these three operations. Each one replaces a maintenance job you would otherwise have to schedule yourself.
“Predictive optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables (Delta Lake and Iceberg) on Databricks”Source: docs.databricks.com
Checkpoint 3 of 6· Exam question
A data engineering team manages dozens of Unity Catalog managed Delta tables that previously required scheduled jobs to run `OPTIMIZE` and `VACUUM` on a nightly cron schedule. Leadership wants to retire these maintenance jobs without losing the performance and storage benefits they provide. Which change lets Databricks take over this maintenance automatically for these tables?
Correct answer: A — Enable predictive optimization at the catalog or schema level so Databricks automatically runs `OPTIMIZE`, `VACUUM`, and `ANALYZE` on the managed tables underneath it.
- A. Predictive optimization is the feature purpose-built for this: once enabled on a catalog, schema, or table, Databricks identifies which managed tables would benefit and automatically queues `OPTIMIZE`, `VACUUM`, and `ANALYZE` as serverless jobs, replacing manual cron-scheduled maintenance.
- B. External tables move Databricks out of managing the underlying files, which removes the automated compaction and retention benefits entirely rather than replacing them; predictive optimization specifically requires Unity Catalog managed tables, not external ones.
- C. Write-time clustering only reorganizes newly written data above certain size thresholds; it does not perform the file compaction, stale-file removal, or statistics collection that scheduled `OPTIMIZE`/`VACUUM`/`ANALYZE` jobs provide, so historical data and storage growth remain unmanaged.
- D. A larger warehouse increases compute cost for every query but does nothing to compact small files, remove tombstoned files, or refresh statistics, so the underlying maintenance problem the team wants automated is left unresolved.
Sources3
3.Turning it on, overriding it, and checking what it did
Predictive optimization is enabled by default for accounts created on or after November 11, 2024, and is being rolled out gradually to older accounts. To use it, your workspace must be on the Premium plan or above in a supported region, you must use SQL warehouses or Databricks Runtime 12.2 LTS or above, and the table must be a Unity Catalog managed table. It doesn't run on external tables or on tables received as an OpenSharing recipient.
The setting is inherited down the hierarchy. All managed tables inherit the account value by default, and you can override it at catalog, schema or table level. An account admin sets the account value in the accounts console. Owners, or users with the MANAGE privilege, set the lower levels in SQL.
Checkpoint 4 of 6· Put it in order
Put the steps for an account admin enabling predictive optimization for the whole account in order
- 1.Navigate to Settings, then Feature enablement
- 2.Select the option you want (for example, Enabled) next to Predictive optimization
- 3.Go to the accounts console
The account-level switch is in the accounts console under Settings > Feature enablement. Catalogs and schemas inherit it unless they have an override.
“Navigate to Settings, then Feature enablement.”Source: docs.databricks.com
ALTER CATALOG [catalog_name] { ENABLE | DISABLE | INHERIT } PREDICTIVE OPTIMIZATION;
ALTER { SCHEMA | DATABASE } schema_name { ENABLE | DISABLE | INHERIT } PREDICTIVE OPTIMIZATION;
ALTER TABLE table_name { ENABLE | DISABLE | INHERIT } PREDICTIVE OPTIMIZATION;Explicit settings take precedence over inherited ones. If you disable predictive optimization at the account level, catalogs or schemas that explicitly enabled it stay enabled. Objects you explicitly disabled before turning it on at account level stay disabled. INHERIT removes the override so the object follows its parent again.
You can also see what predictive optimization decided. DESCRIBE (CATALOG | SCHEMA | TABLE) EXTENDED shows whether it's enabled and whether that value is inherited. On Databricks Runtime 18 LTS and above, the JSON form returns a predictive_optimization_evaluations field with the latest result for COMPACTION, CLUSTERING, AUTO_CLUSTERING_COLUMN_SELECTION and VACUUM. Skip reasons for ANALYZE aren't shown there. In the History tab in Catalog Explorer, operations that ran are labelled Auto and skipped ones Not applied. Results can take up to 24 hours to appear.
DESCRIBE TABLE EXTENDED catalog_name.schema_name.table_name AS JSONCheckpoint 5 of 6· Check yourself
Catalog sales has predictive optimization explicitly enabled. An account admin then disables it at the account level. What happens to managed tables in sales that inherit from it?
An explicit setting on a catalog or schema overrides the account value. Schemas and tables in that catalog inherit from the catalog, so they stay enabled.
“Disabling predictive optimization at the account level does not disable it for catalogs or schemas that have specifically enabled it.”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
A schema in Unity Catalog has predictive optimization enabled, and most of its managed tables inherit that setting. One table in the schema, `finance.ledger_raw`, is extremely write-heavy and the team wants to exclude only this table from automatic maintenance while every other table in the schema keeps running under predictive optimization. Which two statements about accomplishing this are correct? (Select 2)(Select 2)
Correct answers: A, E — A table owner or a user with `MANAGE` privilege on `finance.ledger_raw` can explicitly disable predictive optimization at the table level to override the schema-level setting.; Once a table is excluded from predictive optimization, `OPTIMIZE`, `VACUUM`, and `ANALYZE` must be scheduled manually for that table if maintenance is still desired.
- A. Predictive optimization can be enabled or disabled at the account, catalog, schema, or table level, and table owners or holders of `MANAGE` privilege on that table can set an explicit table-level value that overrides the inherited schema setting.
- B. The enablement hierarchy flows downward from account through catalog and schema to table, with each level inheriting the setting above it unless explicitly overridden, which is exactly the mechanism that lets one table opt out while siblings stay enabled.
- C. Settings are scoped per object; overriding predictive optimization on one table has no effect on any other table in the schema, since each managed table can carry its own explicit setting independent of its siblings.
- D. Granularity goes down to the individual table, so moving the table to a new catalog is unnecessary; a table-level override accomplishes the same exclusion without relocating any data.
- E. Once a table opts out of predictive optimization, Databricks stops automatically queuing maintenance for it, so any ongoing `OPTIMIZE`, `VACUUM`, or `ANALYZE` work for that table would need to be run manually or through a separate scheduled job.
- F. Enabling or disabling predictive optimization is a self-service setting exposed to account admins and object owners with `MANAGE` privilege through the UI, SQL, or API; it does not require opening a support ticket with Databricks.
Sources3
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Enabling predictive optimization keeps an existing Z-ordered table Z-ordered automatically.Why is that wrong?
The OPTIMIZE that predictive optimization runs never runs ZORDER, and it ignores Z-ordered files. To get automatic layout maintenance, move the table to liquid clustering.
Covered in What predictive optimization runs for you
2.Setting delta.deletedFileRetentionDuration to 1 day makes predictive optimization vacuum files after 1 day.Why is that wrong?
Predictive optimization enforces a minimum of 7 days when it runs VACUUM FULL, to protect against data loss and keep recent time travel.
Covered in What predictive optimization runs for you
3.Predictive optimization can be switched on for an external Delta table registered in Unity Catalog.Why is that wrong?
Predictive optimization only supports Unity Catalog managed tables. External tables are excluded because Unity Catalog doesn't control their files.
Covered in Turning it on, overriding it, and checking what it did
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/data-governance/unity-catalog/managed-versus-externalOfficial docs
“External assets: Unity Catalog controls governance only. The underlying file storage lifecycle is controlled by you or an external system.”
↩︎ Who owns the files decides who does the maintenance“Unity Catalog controls both governance (access control, auditing, lineage) and the underlying file storage lifecycle”
↩︎ Key concept - 2.https://docs.databricks.com/aws/en/tables/managedOfficial docs
“Compared to external and foreign tables, managed tables cost less to store and query, maintain and optimize themselves automatically”
↩︎ Who owns the files decides who does the maintenance“For external tables, you must manually delete the files from your storage bucket.”
↩︎ Checkpoint - 3.
“Identifies tables that would benefit from maintenance operations and queues those operations to run.”
↩︎ What predictive optimization runs for you“Your account is billed for this compute using a serverless jobs SKU.”
↩︎ What predictive optimization runs for you“set this property before you enable predictive optimization”
↩︎ What predictive optimization runs for you“All Unity Catalog managed tables inherit the account value by default.”
↩︎ Turning it on, overriding it, and checking what it did“the Auto label indicates the operation ran, and the Not applied label indicates the operation was skipped.”
↩︎ Turning it on, overriding it, and checking what it did“OPTIMIZE does not run ZORDER when executed by predictive optimization.”
↩︎ Exam trap 1“predictive optimization still retains data files for a minimum of 7 days when it runs VACUUM FULL”
↩︎ Exam trap 2“Only Unity Catalog managed tables are supported.”
↩︎ Exam trap 3“OPTIMIZE does not run ZORDER when executed by predictive optimization. On tables that use Z-order, predictive optimization ignores Z-ordered files.”
↩︎ Prediction“Predictive optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables (Delta Lake and Iceberg) on Databricks”
↩︎ Checkpoint“Navigate to Settings, then Feature enablement.”
↩︎ Checkpoint“Disabling predictive optimization at the account level does not disable it for catalogs or schemas that have specifically enabled it.”
↩︎ Checkpoint