What you will be able to do
- Explain what a GDPR/CCPA right-to-be-forgotten request requires and why complete erasure is preferred over obfuscation
- Delete one user's records with DELETE, and a batch of requests with MERGE driven by a control table
- Propagate bronze deletions to materialized views and streaming tables without breaking the pipeline
Key concept
Bronze-first erasure driven by a deletion-request table — Treat every right-to-be-forgotten request as a row in a control table. A scheduled job deletes the matching records from the bronze layer first, and the deletes then flow on to silver and gold. You make the deletion once at the source and let the medallion layers carry it forward, rather than hunting for PII separately in every table.
1.What a right-to-be-forgotten request actually demands
GDPR and CCPA give a customer the right to have all personally identifiable information (PII) about them permanently and completely deleted when they ask. This is often called the "right to be forgotten" (RTBF) or the "right to data erasure". The request also comes with a deadline. You must carry it out within a set period, for example one calendar month. So the job is more than removing some rows. You need a repeatable process that finishes on time and leaves nothing behind that could identify the person.
Your table design can make the job smaller. If user records are keyed on a non-natural key such as user_id and not on a natural key like email, you can sometimes break the link between PII and non-PII data. You delete the PII and keep the anonymous behavioural data. If your design doesn't allow that, you delete the user's records completely, and that is the case the rest of this lesson covers.
The obligation also reaches beyond Delta Lake. GDPR and CCPA apply to data in Kafka topics, raw files and external databases too. A purge strategy that only touches lakehouse tables is incomplete if the same person's data is still sitting in the ingestion queue or the landing zone in cloud storage.
Checkpoint 1 of 4· Check yourself
After your Databricks job has deleted a customer from every bronze, silver and gold table, what else does the GDPR guidance tell you to handle?
The regulations cover all data, not only Delta tables. Upstream systems like Kafka queues and cloud storage files have to be cleaned separately.
“you must also remember to delete data in upstream sources, such as queues and cloud storage”Source: docs.databricks.com
Sources1
2.Deleting in bronze: DELETE for one user, MERGE for a batch
Bronze is where the raw records come in, so it is where erasure starts. Delta Lake supports this with ACID transactions that make point deletes fast even on large data lakes. For a single request, a plain DELETE with a predicate on the user's key is enough. DELETE FROM works only on Delta Lake tables, and its WHERE clause can use subqueries such as IN and EXISTS.
spark.sql("DELETE FROM bronze.users WHERE user_id = 5")Requests rarely come in one at a time. For many records at once, Databricks recommends MERGE. You keep a control table, here called gdpr_control_table, with a user_id column, and insert one row for each user who has asked to be forgotten. The MERGE matches target rows to control-table rows on user_id and deletes every match. This is the scheduled job from the key concept. When the MERGE succeeds, update the control table to record that the request has been processed. That gives you an audit trail showing each request was handled within the deadline.
Checkpoint 2 of 4· Fill the gap
Complete the batch-erasure MERGE so that every target row matching a request in the control table is removed.
spark.sql("""
MERGE INTO target
USING (
SELECT user_id
FROM gdpr_control_table
) AS source
ON target.user_id = source.user_id
WHEN MATCHED THEN ?
""")For an erasure MERGE, the matched clause has to be DELETE. UPDATE SET * would only overwrite the row and INSERT * applies to unmatched rows, so neither removes the user.
Source: docs.databricks.com3.Propagating the delete to silver and gold
Deleting from bronze is only the first step. Once the bronze delete is done, you have to push the change into the silver and gold tables built from it. How you do that depends on the kind of downstream object.
| Downstream object | Behaviour when source rows are deleted | What you must do |
|---|---|---|
| Materialized view | Handles deletions in sources automatically and always returns the correct result (a delete may trigger a full recompute) | Refresh the materialized view and run maintenance so the deletions are fully processed |
| Streaming table (default) | Processes append-only data; a delete in the source breaks the stream | (1) DML delete in the source, (2) DML delete in the streaming table, (3) set skipChangeCommits on the streaming read |
| Streaming table (alternative) | Full refresh clears streaming state and reprocesses everything | Only when historical data is still available and reprocessing is cheap; aged-out upstream data (for example, a Kafka topic) is lost |
Materialized views take the least work. They use incremental computation only when it is cheaper and never at the cost of correctness, so a deleted source row cannot survive in the view. You still have to refresh the view and run maintenance before the deletion is fully processed.
Streaming tables need more care. The recommended pattern is to delete with DML in the source, delete the same rows with DML in the streaming table, and then set skipChangeCommits on the streaming read. That flag tells the stream to skip anything that isn't an insert, so your delete commits stop breaking it. The alternative is a full refresh. It is risky because any upstream source past its retention period, such as a Kafka topic that ages out data after 7 days, cannot be read again. For a sturdier long-term design, stream from the Delta change data feed and handle updates and deletes in your own processing code.
Checkpoint 3 of 4· Put it in order
Put the recommended steps for erasing a user from a streaming table pipeline in order.
- 1.Delete the user's data from the streaming table using DML
- 2.Update the streaming read to use skipChangeCommits
- 3.Delete the user's data in the source Delta table using DML
First the source and then the streaming table are cleaned with DML. After that, skipChangeCommits tells the stream to ignore the non-insert commits those deletes created.
“(2) delete data from the streaming table using DML, and then (3) update the streaming read to use skipChangeCommits”Source: docs.databricks.com
Checkpoint 4 of 4· Exam question
A Unity Catalog managed Delta table `user_profiles` has deletion vectors enabled. To satisfy a GDPR right-to-erasure request, an engineer runs: `DELETE FROM user_profiles WHERE user_id = 'U123'` A subsequent privacy audit finds that the physical row data for `U123` is still present in the underlying Parquet files. What should the team do to permanently remove those physical records?
Correct answer: A — Run `REORG TABLE user_profiles APPLY (PURGE)` to rewrite the affected data files and physically remove rows that deletion vectors had only marked as deleted.
- A. Deletion vectors mark rows as deleted without rewriting the data files that contain them, so the raw record can still exist on disk after a `DELETE`. `REORG TABLE ... APPLY (PURGE)` rewrites those files to physically remove the marked rows, which is the documented step for permanently purging deletion-vector-enabled tables.
- B. `VACUUM` only removes data files that are no longer referenced by any live table version; it does not rewrite files that are still referenced just because deletion vectors mark some of their rows as removed. Files holding the target row remain referenced and untouched, so this does not purge the physical record.
- C. Deletion vectors record which rows are logically deleted the first time a matching `DELETE` runs; there is no mechanism by which a second identical delete against an already-empty result set changes file-level storage. Repeating the statement has no additional effect on the physical files.
- D. Turning off the `delta.enableDeletionVectors` property changes how future writes are handled but does not trigger an automatic rewrite of existing data files or retroactively purge previously vector-marked rows. A separate rewrite operation such as `REORG` or `OPTIMIZE` is still required to remove the physical data.
Sources1
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A streaming table picks up deletes from its Delta source the way a materialized view does.Why is that wrong?
Only materialized views handle source deletions automatically. A streaming table expects append-only input, so a delete in the source breaks the stream unless you delete with DML in both tables and set skipChangeCommits.
Covered in Propagating the delete to silver and gold
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/ldp/gdprOfficial docs
“deletion requests must be executed during a specified period (for example, within one calendar month)”
↩︎ What a right-to-be-forgotten request actually demands“you can delete PII data, which leaves non-PII data in place”
↩︎ What a right-to-be-forgotten request actually demands“When deleting a large number of records together at one time, we recommend using the MERGE command.”
↩︎ Deleting in bronze: DELETE for one user, MERGE for a batch“After this MERGE command is successful, update the control table to confirm that the request has been processed.”
↩︎ Deleting in bronze: DELETE for one user, MERGE for a batch“You must refresh a materialized view and run maintenance to ensure that deletions are completely processed.”
↩︎ Propagating the delete to silver and gold“This flag indicates that the streaming table should skip anything other than inserts, such as updates or deletes.”
↩︎ Propagating the delete to silver and gold“a Kafka topic that ages out data after 7 days) won't be processed again, which could cause data loss”
↩︎ Propagating the delete to silver and gold“stream from the change feeds of Delta tables instead, and handle updates and deletes in your processing code”
↩︎ Propagating the delete to silver and gold“deleting data in the bronze layer first, driven by a scheduled job that queries a table of deletion requests”
↩︎ Key concept“Streaming tables process append-only data when they stream from Delta table sources.”
↩︎ Exam trap 1“the safest option is complete erasure because, in practice, eliminating the risk of reidentification often requires a complete deletion of PII data”
↩︎ Prediction“you must also remember to delete data in upstream sources, such as queues and cloud storage”
↩︎ Checkpoint“Any other operation, such as updating or deleting a record from a streaming source, is not supported and breaks the stream.”
↩︎ Prediction“(2) delete data from the streaming table using DML, and then (3) update the streaming read to use skipChangeCommits”
↩︎ Checkpoint - 2.
“This statement is only supported for Delta Lake tables.”
↩︎ Deleting in bronze: DELETE for one user, MERGE for a batch