Free Databricks Certified Data Engineer Professional Sample Questions

Correct answer: B — In the task definition, include a `parameters` map with a key for the date and a default value, like `parameters: { 'processing_date': '2023-01-01' }`.

• A. Incorrect. Jinja templating in Databricks Asset Bundles is used for substituting variables from the `bundle.variables` block into resource definitions at deployment time. It is not the correct mechanism for defining dynamic, runtime parameters that are passed to a job's code during execution.
• B. Correct. This is the standard and intended method for defining job parameters in a Databricks Asset Bundle. The `parameters` map within a task definition (e.g., `notebook_task`) explicitly declares the parameters the job accepts. Providing a default value is a best practice, and this structure is designed to be easily overridden at runtime via the API, CLI, or UI when the job is triggered.
• C. Incorrect. The top-level `bundle.variables` block is for parameterizing the bundle's infrastructure and resource definitions (e.g., different cluster IDs for dev vs. prod), which are resolved at deployment time. This is distinct from job task parameters, which are values passed to the code at runtime.
• D. Incorrect. While it might be technically possible to pass parameters via the API without declaring them in the YAML, this is against best practices. It undermines the Infrastructure as Code (IaC) principle of having a single, version-controlled source of truth for the job's configuration. For clarity, maintainability, and discoverability, all expected job parameters should be explicitly defined in the `databricks.yml` file.

Correct answer: B — A recent change to the init script has not been applied because the cluster was not fully restarted after the script was updated.

• A. Incorrect. The problem states that the init script's logs confirm a successful `pip install`. This indicates the script was successfully accessed, read, and executed by the cluster nodes, which rules out a permissions issue with the service principal.
• B. Correct. This is the most probable cause. In Databricks, any changes made to a cluster-scoped init script require a full cluster restart to be reliably applied to all existing nodes. A resize operation adds or removes worker nodes but does not constitute a full restart. If an administrator updated the init script (e.g., to add a new library) and then resized the cluster without restarting it first, an inconsistent state would occur. The existing driver and worker nodes would still be running with the old environment, while new nodes might get the updated script. If the job code requires the new library, it would fail with a `ModuleNotFoundError` when running on a node (like the driver) that has not been restarted to pick up the script change.
• C. Incorrect. A cluster-scoped init script, by definition, is designed to run on every node in the cluster, including the driver and all worker (executor) nodes. For the library to be installed only on the driver, the script would need to contain explicit logic to do so (e.g., `if [ "$DB_IS_DRIVER" == "true" ]`). While possible, this is an anti-pattern and is less likely to be the root cause than a common operational error like forgetting to restart after a script update.
• D. Incorrect. A `ModuleNotFoundError` indicates that the Python interpreter was unable to locate the specified module. A library version conflict with a built-in library would typically manifest as a different error after the module is found and loaded, such as an `ImportError`, `AttributeError`, or another runtime exception, not an inability to find the module itself.

Correct answer: C — To list the columns from the source stream that should be inserted or updated in the target table, allowing for exclusion of metadata columns.

• A. Incorrect. The primary keys for the merge operation are defined using the `KEYS` clause, not `COLUMNS`. The `COLUMNS` clause is used to select which source columns are propagated to the target table.
• B. Incorrect. The sequencing of out-of-order events is controlled by the `SEQUENCE BY` clause, which specifies the column that determines the correct chronological order. The `COLUMNS` clause does not handle event ordering.
• C. Correct. The optional `COLUMNS` clause explicitly lists the source stream columns to be inserted or updated in the target table. This is useful for excluding metadata columns, such as `_rescued_data` from Auto Loader, that are not intended for the target schema.
• D. Incorrect. Column renaming is a transformation that should be performed in the source query, typically using `AS` aliases in the `SELECT` statement. The `COLUMNS` clause only specifies which columns to include by their source name and does not perform renaming.

Correct answer: D — Specify the required libraries in the task's `dependent_libraries` setting in the job definition, pointing to a requirements.txt file or PyPI packages.

• A. Incorrect. While using `%pip install pandas==1.5.3 great_expectations==0.17.0` in the first notebook cell works, it is not the most robust method for production. This approach adds installation overhead to every run, can lead to failures due to network issues with PyPI, and tightly couples dependency management with the application code, which is not a best practice.
• B. Incorrect. A cluster init script installs libraries for the entire cluster, not just a specific task. This lacks the necessary granularity and can cause dependency conflicts if the same cluster is used by other jobs with different requirements. It also increases cluster startup time.
• C. Incorrect. Packaging the notebook and its dependencies into a Python wheel file and running the job as a `spark_python_task` is an unnecessarily complex workflow for this problem. This approach fundamentally changes the task from a notebook-based one to a Python script-based one, which is not what is being asked and introduces significant overhead for simply managing libraries.
• D. Correct. This is the recommended and most robust method. Specifying the required libraries in the task's `dependent_libraries` setting in the job definition declaratively manages the environment. This ensures that the exact versions are installed in an isolated environment for that specific task, guaranteeing reproducibility and protecting against upstream changes without modifying the notebook code or affecting the entire cluster.

Correct answer: C — Add an `If/else condition` task that evaluates `{{tasks.check_staleness_task.values.is_stale}} == 'true'` and place `archive_notebook.py` as a child of the `true` branch.

• A. The `All done` condition triggers execution regardless of success or failure, but not based on the task's output value. The archival notebook would run even when `is_stale` is `false`, which is not the desired behavior. Task‑value‑based decisions require an `If/else condition` task.
• B. `At least one succeeded` ensures the upstream task completed successfully, but it does not evaluate the task's return value. The notebook would be executed even when `is_stale` is `false`, because the upstream task itself can succeed while outputting `false`. This does not satisfy the requirement to run only when staleness is `true`.
• C. This is the intended method for conditional branching based on a task value. Databricks Jobs allow `If/else condition` tasks to define boolean logic using task variables. Because `dbutils.jobs.taskValues.set` serializes a boolean to the string `"true"`, the comparison `== 'true'` correctly triggers the `true` branch, where the archival notebook can be executed.
• D. `Run if` dependencies only accept predefined status‑based conditions (e.g., `All succeeded`, `At least one failed`). They cannot evaluate custom expressions against task values. To inspect the value of `is_stale`, an `If/else condition` task must be used as shown in option 3.

Correct answers: A, B — Set `.option("cloudFiles.useNotifications", "true")` in the Auto Loader readStream configuration.; Assign appropriate cloud provider permissions (e.g., IAM role, Event Grid Contributor) to the Databricks cluster or service principal to allow automated resource creation.

• A. Correct. To enable file notification mode in Auto Loader, you must set the `.option("cloudFiles.useNotifications", "true")` configuration. This tells Auto Loader to use a pub/sub notification model (such as AWS SNS/SQS or Azure Event Grid) instead of recursive directory listing.
• B. Correct. File notification mode requires the automatic setup and management of cloud resources (e.g., queues, topics, and event subscriptions). The cloud identity (IAM role or Service Principal) used by Databricks must have sufficient permissions to create and configure these resources on your behalf.
• C. Incorrect. The source format for Auto Loader is always "cloudFiles". You switch between listing and notification modes using the cloudFiles options, not by changing the read format to "cloudQueue".
• D. Incorrect. The cloudFiles.fetchParallelism option is specifically used to optimize the default directory listing mode by increasing the number of threads used for discovery. It is not used for file notification mode.
• E. Incorrect. VACUUM is a Delta Lake command used for data retention management (removing files no longer referenced by the Delta log). It has no relation to the ingestion configuration or the cloud provider's notification services.

Correct answer: C — Explicitly `select()` only the 10 required columns from the source DataFrame immediately after the `.load()` call to enable column pruning.

• A. Incorrect. Converting the source files to Delta format before ingestion adds an extra processing step and associated cost. While Delta Lake offers significant read performance benefits for subsequent queries, it does not solve the immediate problem of the initial job reading too many columns from the original ORC source. The most direct solution is to optimize the read operation itself.
• B. Incorrect. Applying a `filter()` operation enables predicate pushdown and partition pruning, which are powerful techniques for reducing the number of *rows* or *files* scanned. However, the bottleneck described in the scenario is reading too many *columns* (100 instead of 10), not too many rows. This option addresses a different, though also important, type of performance issue.
• C. Correct. This is the most effective solution because it directly enables column pruning. According to Databricks best practices, selecting only the necessary columns as early as possible allows the Catalyst Optimizer to push this operation down to the data source. For columnar formats like ORC, this means only the data for the 10 required columns will be read from storage, significantly reducing I/O, network traffic, and memory usage.
• D. Incorrect. Increasing cluster resources is a brute-force approach that may speed up the job but does not fix the underlying inefficiency. The root cause is that the job is performing unnecessary work by reading 90 extra columns. The most effective and cost-efficient optimization is to eliminate this unnecessary work rather than adding more compute power to handle it.

Correct answer: B — Configure .option("cloudFiles.rescuedDataColumn", "_rescued_data") to capture unparsed or type-mismatched data into a specific column.

• A. Incorrect. Setting `.option("mode", "DROPMALFORMED")` causes corrupt records to be silently dropped, which violates the requirement that no data is dropped. Additionally, it does not retain the problematic data in the target Delta table for debugging.
• B. Correct. Configuring `.option("cloudFiles.rescuedDataColumn", "_rescued_data")` captures unparsed or type-mismatched data into a dedicated column, allowing the pipeline to continue without failure. This ensures all original data is preserved in the Delta table for later inspection.
• C. Incorrect. Setting `.option("badRecordsPath", "<path>")` redirects entire JSON files containing malformed records to an external quarantine location, which means the problematic data is not retained in the target Delta table. The requirement explicitly states that data must be kept in the target table for debugging.
• D. Incorrect. Using `.option("cloudFiles.schemaEvolutionMode", "rescue")` does not evolve the schema by adding new columns for unexpected data types; that behavior belongs to the `addNewColumns` mode. The `rescue` mode instead captures mismatched data into a single rescued data column, which does not match the described functionality.

Correct answer: B — A Pandas UDF (Vectorized UDF) of type `SCALAR`, which processes data in batches using Apache Arrow.

• A. Incorrect. A standard Python UDF processes data row-by-row. This involves significant serialization and deserialization overhead as data is moved between the Spark JVM and the Python process for each individual row. This method is notoriously slow and inefficient for large DataFrames.
• B. Correct. Pandas UDFs, also known as Vectorized UDFs, are the most performant option for this scenario. They use Apache Arrow for zero-copy, efficient data transfer between the JVM and Python. Data is processed in batches (as Pandas Series), which dramatically reduces the serialization overhead. This batch processing also allows for the use of vectorized operations within the Python function, which is ideal for computationally intensive tasks and leveraging optimized third-party libraries.
• C. Incorrect. Using `forEach` is an action, not a transformation, and it applies a function to each row without returning a new DataFrame. It forces row-by-row execution and does not leverage Spark's Catalyst optimizer or parallel processing capabilities for transformations, leading to extremely poor performance on large datasets.
• D. Incorrect. Collecting a large column to the driver node is a major anti-pattern in distributed computing. It creates a severe bottleneck, can easily cause driver OutOfMemoryErrors, and completely defeats the purpose of parallel processing. This approach is not scalable and will fail on very large DataFrames.

Correct answer: B — The alignment of the window function's `PARTITION BY` and `ORDER BY` clauses with the table's Z-ORDER keys makes the required data shuffle more efficient.

• A. Incorrect. Changing `ROWS BETWEEN` to `RANGE BETWEEN` alters the logical definition of the window frame but does not inherently leverage the physical data layout for optimization. While both can define a window, this change is not the primary mechanism by which Z-ORDERING would improve query performance.
• B. Correct. Window functions with `PARTITION BY` and `ORDER BY` require a shuffle to group and sort data. According to Databricks documentation, while Z-ORDERING does not eliminate the shuffle, having data physically co-located and pre-sorted on disk makes the shuffle operation significantly more efficient by reducing the volume of data that needs to be moved across the network and sorted by the executors.
• C. Incorrect. In modern distributed processing engines like Spark, window functions are highly optimized and are the recommended approach for calculations like moving averages. A self-join to achieve the same result would be more complex to write and almost certainly less performant than the native window function.
• D. Incorrect. This statement is a common misconception. According to Databricks documentation, window functions that partition and order data are wide transformations that inherently require a shuffle. Z-ORDERING improves data locality and can reduce the amount of data read from storage, which makes the shuffle more efficient, but it does not eliminate the shuffle operation itself.

Correct answer: C — In the hub workspace, create three separate shares (e.g., `finance_share`, `marketing_share`, `sales_share`), add the relevant tables to each, and then assign the corresponding spoke workspace's Metastore ID as a recipient to its specific share.

• A. Incorrect. This describes open sharing, which relies on manually managed bearer tokens. This approach is less secure and scalable compared to Databricks-to-Databricks sharing, introduces operational overhead of distributing and rotating tokens, and grants full access to all shared tables, violating least privilege.
• B. Incorrect. While this uses Databricks-to-Databricks sharing, creating a single monolithic share for all data violates the principle of least privilege. It would grant all spoke workspaces identical access to the entire curated dataset, not just the subsets they are authorized to see.
• C. Correct. This approach perfectly aligns with all requirements. Creating separate, purpose-built shares in the central hub workspace allows for granular, centrally managed access control. Assigning each spoke's Metastore ID as a recipient uses the secure, tokenless Databricks-to-Databricks sharing protocol, enforcing least privilege by ensuring each spoke only receives the data it needs, making it both highly secure and scalable.
• D. Incorrect. Lakehouse Federation is designed for querying data in external data sources (like PostgreSQL, MySQL, Redshift) from Databricks, not for sharing data between Databricks workspaces. Delta Sharing is the purpose-built feature for this use case. Furthermore, this approach would require configuration in each spoke, violating the central management requirement.

Correct answer: B — It is a configuration profile containing the sharing server endpoint and a bearer token that the recipient's client uses to authenticate and fetch temporary data URLs.

• A. Incorrect. The credential file does not contain pre-signed URLs. Instead, it provides the sharing server endpoint and a bearer token, which the recipient's client uses to authenticate and dynamically fetch temporary pre-signed URLs for data access.
• B. Correct. The credential file is a configuration profile that includes the sharing server endpoint and a bearer token. The recipient's client uses these to authenticate with the server and obtain temporary data URLs, such as pre-signed URLs, for accessing the underlying Parquet files.
• C. Incorrect. The credential file is used for authentication, not decryption. It contains a bearer token, not a private key, and the Delta Sharing protocol does not rely on public/private key encryption for recipient access.
• D. Incorrect. The credential file does not provide a direct connection string to the provider's Unity Catalog metastore, as that would pose a security risk. It only grants access to the sharing server, which acts as a secure proxy for data retrieval.

Correct answer: D — Use `{{priority}}` and `{{error_count}}` in the custom template, referring to the column aliases from the query result.

• A. Incorrect. Databricks SQL Alert templates are designed to be dynamic and can incorporate values directly from the alert's query results. Hardcoding the message is unnecessary and defeats the purpose of data-driven alerting.
• B. Incorrect. This syntax, which attempts to access results by positional index, is not valid for Databricks SQL Alert templates. The correct method for accessing query results is by referencing the column names or aliases.
• C. Incorrect. The `{{QUERY_RESULT_VALUE}}` variable is a valid placeholder, but it is specifically for alerts where the query returns a single scalar value. It cannot be used to access multiple, distinct columns like `error_count` and `priority`. Creating a second alert is inefficient and unnecessary as a single query can provide all the required information.
• D. Correct. Databricks SQL Alerts support custom notification templates using a templating language. To include values from the query result, you wrap the column aliases in double curly braces. For this scenario, the custom template message would be configured as `{{priority}} priority alert: {{error_count}} errors found.` to dynamically generate the desired notification.

Correct answer: C — The amount of data that could not fit in memory and was written to disk.

• A. Incorrect. 'Spill Size' is unrelated to the cluster's cache. Caching is an optimization technique to persist data in memory for faster access. Spilling, in contrast, is a performance penalty that occurs when an operation runs out of memory and must temporarily write intermediate data to disk.
• B. Incorrect. Broadcasting is a specific join strategy where a small DataFrame is sent to all executor nodes. This is a distinct operation tracked by different metrics and is not related to spilling, which is a mechanism to handle memory overflow during operations like joins, sorts, or aggregations.
• C. Correct. The 'Spill Size' metric quantifies the amount of data that an operator could not hold in the executor's RAM and had to 'spill' to local disk storage. A non-zero spill size is a critical indicator of memory pressure and a significant performance bottleneck, as disk I/O is substantially slower than in-memory processing. Identifying operators with high spill size is a key step in query optimization.
• D. Incorrect. The total size of the output data is a different metric, often represented by the number of output rows or the size of data written to the next stage. 'Spill Size' exclusively measures the intermediate data that was offloaded to disk during the execution of the operator due to insufficient memory, not the final output of the operator.

Correct answer: A — Enable the option to send a notification when the alert status changes from `TRIGGERED` back to `OK`.

• A. Databricks SQL Alerts track state transitions. When a query result returns to a state that no longer meets the alert's trigger criteria, the status transitions from TRIGGERED back to OK. Enabling or configuring notifications for this state transition ensures the operations team receives a resolution or follow-up message when the pipeline catches up.
• B. Creating a second alert is redundant and creates additional maintenance overhead. The built-in state management of a single Databricks SQL Alert already handles both the 'Triggered' (problem) and 'OK' (recovery) states.
• C. Rearm settings (or notification frequency) control how often an alert sends notifications while it remains in a TRIGGERED state. It does not define the logic for sending a resolution message, nor does an empty result set equate to a 'resolved' state for a threshold-based watermark alert.
• D. Custom templates are used to format the visual appearance and content of the notification message. They do not control the logic or timing of when the Databricks SQL Alert service decides to emit a notification.

Correct answer: C — The `WHERE` clause applies a function (`to_date`) to the partition column, which prevents the optimizer from being able to push the filter down to the metadata level.

• A. Incorrect. Partition pruning is a fundamental optimization in Spark SQL and Databricks, supported across all modern SQL Warehouse versions. The optimizer can eliminate partitions when a filter on the partition column is present, so lack of support is not the cause.
• B. Incorrect. Stale table statistics can affect join strategies and cardinality estimates, but they do not prevent partition pruning. The optimizer can still prune partitions based on a direct filter predicate on a partition column, even without up-to-date statistics.
• C. Correct. Applying a function like `to_date()` to the partition column in the `WHERE` clause prevents the optimizer from pushing the filter down to the metadata level. The function must be evaluated on every row, so the optimizer cannot use the raw partition key for pruning. To enable pruning, apply the function to the literal value instead, e.g., `WHERE partition_col = '2023-10-26'` (if the column is a string) or `WHERE partition_col = date('2023-10-26')` (if the column is a date type).
• D. Incorrect. Partition pruning is a metadata-level operation performed during query planning and does not depend on driver memory to hold partition metadata. Insufficient driver memory may cause other issues, but it does not cause the optimizer to fall back to a full scan due to inability to load the partition list.

Correct answer: A — Add two separate notification blocks in the job's settings: one for the `On failure` event with the lead's email, and another for the `On failure` event with the Slack destination.

• A. Correct. The Databricks Workflows UI requires a separate notification block for each destination, even for the same trigger event. To notify both an email address and a pre-configured Slack destination upon failure, the engineer must add two distinct notification blocks for the 'On failure' event. This is the standard and correct implementation pattern within Databricks Jobs.
• B. Incorrect. This method introduces unnecessary complexity, latency, and potential points of failure by requiring a custom notebook to handle notifications. The built-in, declarative notification system in Databricks Workflows is the standard, more reliable, and preferred method for sending alerts.
• C. Incorrect. This is an unreliable approach because many job failures, such as code errors or data quality issues, do not result in a timeout. The job could fail quickly, and this notification would never be triggered. The 'On failure' event is the correct trigger to capture all failure scenarios.
• D. Incorrect. The Databricks Workflows UI does not support adding multiple destinations of different types (e.g., an email address and a Slack destination) within a single notification configuration block. To send alerts to multiple destinations for the same event, separate notification blocks must be created for each one.

Correct answer: C — Query `system.access.table_lineage`, filter where `created_by` is the service principal and the event occurred in the last 30 days, and then select the distinct `source_table_full_name`.

• A. Incorrect. This approach violates the requirement to avoid parsing SQL text. Extracting table names from `query_text` using regex is complex, error-prone, and inefficient, especially for queries with CTEs, subqueries, and aliases.
• B. Incorrect. The audit log is not designed for direct lineage tracking. Parsing the `request_params` JSON field is unreliable, violates the 'no parsing' constraint, and may not capture every table read within a single multi-table query.
• C. Correct. The `system.access.table_lineage` table is purpose-built for this task. It automatically captures read and write operations at the table level, providing structured, accurate data without any text parsing.
• D. Incorrect. The `system.billing.usage` table provides no information about which specific data objects were accessed. It only tracks compute usage (DBUs) and costs, making it impossible to infer table access.

Correct answer: B — Remove the `ingest_date` partition and implement Liquid Clustering on `ingest_date`, `device_id`, and `location_id`.

• A. Incorrect. Partitioning by high-cardinality columns like `device_id` and `location_id` is an anti-pattern. According to Databricks documentation, this leads to the 'Small Files Problem' by creating a massive number of small partitions, which increases metadata overhead and slows down both reads and writes.
• B. Correct. Databricks strongly recommends Liquid Clustering for all new tables as a replacement for partitioning and Z-ordering. It is ideal for tables filtered by high-cardinality columns. By removing the partition and clustering on all three columns, the data layout is optimized for the described query patterns without the drawbacks of traditional partitioning.
• C. Incorrect. While Z-ordering on `device_id` and `location_id` is a valid technique that would improve performance, Liquid Clustering is the modern, recommended best practice that replaces both partitioning and Z-ordering. For a comprehensive data layout strategy, migrating to Liquid Clustering is the superior approach.
• D. Incorrect. Liquid Clustering is not compatible with traditional table partitioning. A table must use one or the other. The documentation explicitly states that Liquid Clustering replaces partitioning and Z-ordering, so they cannot be used together on the same table.

Correct answer: B — 'spill size (memory)' and 'spill size (disk)' would indicate sorting issues, while 'shuffle read time' and 'shuffle bytes read' would indicate shuffling issues.

• A. Incorrect. 'Number of output partitions' and 'number of files read' are not metrics of the join operator itself; they relate to downstream parallelism and upstream scanning, respectively. 'Records read' indicates data volume but does not differentiate between time spent sorting versus shuffling.
• B. Correct. 'Spill size (memory)' and 'spill size (disk)' directly indicate sorting bottlenecks, as spills occur when data exceeds memory during sorting. 'Shuffle read time' and 'shuffle bytes read' measure the network transfer cost, clearly isolating shuffling issues.
• C. Incorrect. 'Peak execution memory' and 'total time in GC' reflect general memory pressure but do not specifically distinguish sorting from shuffling bottlenecks. 'Shuffle write time' only partially captures network transfer and does not isolate the join's internal sorting cost.
• D. Incorrect. 'Scan time' and 'number of rows filtered' are metrics from upstream scan and filter operations, not from the SortMergeJoin operator. 'Shuffle fetch wait time' indicates network delays but does not help assess sorting performance.

Correct answer: B — Tokenization, as it is designed to be reversible via a secure lookup/vault system.

• A. Incorrect. Hashing is a one-way cryptographic function that creates a unique, fixed-length string from an input. By design, it is not reversible, meaning the original `patient_medical_record_number` cannot be recovered from the hashed value. This violates the critical requirement to be able to map back to the original data.
• B. Correct. Tokenization is the process of replacing sensitive data with a non-sensitive equivalent, referred to as a 'token'. The original data is stored separately in a secure 'vault' that maps the token back to the original value. This method is designed to be reversible under controlled circumstances, perfectly matching the requirement to use a temporary identifier and maintain a secure lookup system for potential re-identification.
• C. Incorrect. Suppression involves completely removing the identifier from the dataset. This is a form of anonymization, but it makes it impossible to link the data back to the original record, thus failing the requirement to maintain a mapping for future lookups.
• D. Incorrect. Data masking is a general term for obscuring data, often applied dynamically at query time. While some masking techniques can be reversible, the specific process described—replacing data with a random token and maintaining a separate, secure vault for mapping—is precisely the definition of tokenization, making it the most accurate answer.

Correct answer: C — Suppression

• A. Incorrect. Generalization is a data anonymization technique that reduces the precision or granularity of data to prevent identification. For example, replacing an exact age with an age range (e.g., 34 becomes 30-39) or a specific zip code with a broader city. The scenario describes removing entire records, not making them less specific.
• B. Incorrect. Tokenization involves replacing sensitive data elements with non-sensitive, non-reversible placeholders called tokens. This is commonly used for data like credit card numbers. The described technique is about removing records, not substituting data with tokens.
• C. Correct. Suppression is the practice of removing or omitting data to protect privacy. This can involve removing specific fields or, as in this scenario, removing entire records that could lead to the re-identification of individuals. Removing all records for departments with fewer than five employees is a classic example of record suppression to ensure k-anonymity.
• D. Incorrect. Hashing is a cryptographic technique that transforms data into a fixed-length, non-reversible string of characters. It is primarily used for data integrity verification and securing sensitive data like passwords, not for removing records based on group size to protect privacy in an analytical dataset.

Correct answer: B — A two-step job that first executes `DELETE FROM user_activity WHERE event_date < date_sub(current_date(), 1825);` and then runs the `VACUUM user_activity` command.

• A. Incorrect. Overwriting a multi-terabyte table is extremely inefficient and costly. It requires rewriting all data, including the records that are not being deleted, which is unnecessary for this type of targeted deletion.
• B. Correct. This is the most efficient and complete process. The `DELETE` statement is ACID-compliant and leverages partition pruning on the `event_date` column for a highly efficient logical deletion. According to Databricks documentation, the subsequent `VACUUM` command is required to physically remove the old data files from cloud storage, thereby permanently deleting the data and reclaiming space.
• C. Incorrect. While this command would logically delete the correct records, it is highly inefficient. Filtering on `event_timestamp` instead of the partition column `event_date` prevents the query optimizer from using partition pruning, forcing a full scan of the table's data.
• D. Incorrect. The `VACUUM` command does not logically delete records based on a condition. It only cleans up data files that are no longer referenced by the Delta table's transaction log and are older than the retention period. It is a necessary second step after a `DELETE` operation, but it cannot enforce the retention policy on its own.

Correct answer: D — Create a centralized 'deletion_requests' Delta table. An automated, scheduled job reads new requests from this table, iterates through a list of target tables to execute parameterized `DELETE` operations, and logs the outcome of each deletion back to the request table.

• A. Incorrect. Anonymizing data by setting columns to NULL or generic values does not purge the records, which is required for GDPR's 'Right to be Forgotten'. This approach leaves the original rows in place and may still allow re-identification, failing to meet the deletion mandate.
• B. Incorrect. Using a streaming pipeline for ad-hoc deletion requests introduces unnecessary complexity and cost. 'Right to be Forgotten' requests are typically infrequent and batch-oriented, making a scheduled batch job more appropriate and manageable.
• C. Incorrect. A manual notebook process is error-prone, not scalable, and lacks systematic auditability. As the number of tables or requests grows, this approach becomes unmanageable and does not provide a robust, automated compliance solution.
• D. Correct. A centralized deletion_requests table provides a clear, auditable log of all requests and their outcomes. An automated scheduled job ensures the process is repeatable, scalable, and not dependent on manual intervention. For full compliance, this design should be augmented with periodic VACUUM operations to physically remove data files, and if deletion vectors are enabled, REORG TABLE ... APPLY (PURGE) to commit deletions.

Correct answer: A — The row filter function prevents the query optimizer from performing partition pruning.

• A. Correct. When a row filter function performs a lookup against another table, the query optimizer cannot predict the function's outcome during planning. Because the filter logic is opaque, the optimizer cannot determine which partitions can be safely skipped, disabling partition pruning. This forces a full scan of all partitions, causing the observed performance degradation.
• B. Incorrect. Z-ORDERING co-locates related data within files to improve data skipping, but it does not affect partition pruning. Since the dimension table is small, its physical layout has negligible impact on performance. The bottleneck is the full scan of the large partitioned table, not the lookup against the small dimension table.
• C. Incorrect. While non-deterministic functions can prevent certain optimizations like result caching, the core issue here is the failure of partition pruning. Even a deterministic user-defined function would be opaque to the query optimizer, preventing it from reasoning about the filter's output and thus still blocking partition pruning.
• D. Incorrect. Unity Catalog row filters are fully compatible with partitioned tables and do not inherently require a full table scan. The performance issue arises from the specific filter function's interaction with the query optimizer, not from any incompatibility with partitioned tables.

Correct answer: B — The search index for Unity Catalog metadata has not yet been updated to include the recently added comments.

• A. This option is incorrect. Unity Catalog's global search functionality is designed to be comprehensive and indexes not only the names of catalogs, schemas, and tables but also other metadata such as column names, comments, and tags. This allows users to discover data assets based on their descriptions and context.
• B. This is the most likely reason. The Unity Catalog search index is updated asynchronously. While this process is typically fast (often within minutes), a delay in the indexing process is the most plausible technical explanation among the choices for why recently added comments would not appear in search results.
• C. This option is incorrect. In Unity Catalog, the `SELECT` privilege on a table is sufficient to allow a user to view all of its metadata, including table and column comments. While the `BROWSE` privilege allows for asset discovery, having `SELECT` already grants the necessary permissions to see the metadata that would be surfaced by search.
• D. This option is incorrect. A key feature of the Databricks UI is the global search bar, which is designed to index and search metadata like comments to improve data discovery. While the Information Schema can be queried using SQL to retrieve comments, it is not the only, nor the primary, method for interactive searching.

Correct answers: A, B — User A executes `ALTER TABLE enterprise.marketing_data.campaigns OWNER TO marketing_team;`.; User B executes `ALTER TABLE enterprise.marketing_data.campaigns OWNER TO marketing_team;`.

• A. Correct. User A is the original owner of the table because they created it. In Unity Catalog, the owner of an object has the permission to transfer that ownership to another principal (user or group) using the `ALTER TABLE ... OWNER TO` command.
• B. Correct. A Metastore Admin has the highest level of privilege and can manage or change the ownership of any object within the metastore, regardless of who created it.
• C. Incorrect. `ALL PRIVILEGES` grants all permissions available for that object type (like SELECT and MODIFY), but in Unity Catalog, it does not grant ownership. Ownership is required to perform DDL operations like DROP or ALTER TABLE.
• D. Incorrect. The `MODIFY` privilege is strictly for Data Manipulation Language (DML) operations such as INSERT, UPDATE, DELETE, and MERGE. It does not provide the authority to perform Data Definition Language (DDL) operations like changing the schema or dropping the table.
• E. Incorrect. While transferring ownership of the schema would allow the group to manage the table, the question explicitly states that the solution should not give them ownership of the entire schema.

Correct answers: A, B — Revoke the `SELECT` privilege on `sales_catalog` from the `analysts` group.; Grant `SELECT` on the specific schemas (excluding `hr_confidential`) to the `analysts` group.

• A. This step is necessary because the `SELECT` privilege granted at the catalog level is inherited by all current and future schemas, including `hr_confidential`. Revoking it removes all inherited `SELECT` access, allowing the administrator to then regrant access only to the appropriate schemas.
• B. After revoking the broad catalog-level privilege, this step restores `SELECT` access only to the allowed schemas. Note that for the analysts to successfully query tables, they must also have `USE CATALOG` on `sales_catalog` and `USE SCHEMA` on each of these schemas. These usage privileges are typically already in place or must be granted concurrently.
• C. Unity Catalog does not support the `DENY` statement. Access control relies solely on granting and revoking privileges. To restrict access, you must revoke the parent privilege and then grant only the desired subset.
• D. In Unity Catalog, you cannot revoke a privilege on a child securable object if it was granted on a parent. Since `SELECT` was granted on the catalog, the privilege must be revoked at the catalog level, not the schema level.
• E. Ownership of an object grants all privileges to the owning principal, but it does not affect privileges inherited by other principals from a parent grant. The analysts group would still retain `SELECT` on `hr_confidential` through the catalog-level grant regardless of who owns the schema.

Correct answer: C — Use the 'Run now with different parameters' option and provide a JSON payload like `{"report_date": "2023-11-15"}`.

• A. Incorrect. The 'Run now' button executes the job using its default parameters. In this scenario, the default is the current date, so it would not run for the missed day of '2023-11-15'. Databricks jobs do not automatically detect and rerun failed historical dates with a standard 'Run now' trigger.
• B. Incorrect. While this approach would work, it is inefficient and creates unnecessary job management overhead. Cloning a job and hardcoding parameters defeats the purpose of having a flexible, parameterized job and is not the intended or best-practice method for this scenario.
• C. Correct. The 'Run now with different parameters' feature in the Databricks Jobs UI is specifically designed for this use case. It allows the engineer to override the default parameters for a single, ad-hoc execution by providing a JSON payload. This triggers the job for the specified `report_date` without altering the job's saved default configuration for future scheduled runs.
• D. Incorrect. Modifying the underlying notebook code to hardcode the date is a poor practice. It bypasses the job's parameterization, makes the code less reusable, and introduces a high risk of forgetting to revert the change, which would disrupt subsequent automated runs.

Correct answer: A — DABs allow data teams to define, version, and deploy both the infrastructure (e.g., jobs, DLT pipelines) and the source code as a single cohesive unit.

• A. Correct. Databricks Asset Bundles (DABs) represent a shift toward Infrastructure as Code (IaC). While Git Folders focus on synchronizing source code and notebooks, DABs allow developers to define the entire application stack—including compute configurations, Job schedules, Delta Live Tables pipelines, and source code—within a single YAML-based configuration. This ensures that the environment and the code are deployed and versioned together as a single unit.
• B. Incorrect. DABs are a deployment and configuration tool, not a local execution engine for unit tests. While they can be integrated into CI/CD pipelines that run tests, the execution of those tests still typically requires a Databricks workspace (via the Databricks CLI or SDK) or a specialized local testing framework.
• C. Incorrect. DABs do not handle Git-level operations like merge conflict resolution. Conflict resolution remains the responsibility of the developer using Git version control tools prior to the deployment phase.
• D. Incorrect. Databricks Asset Bundles do not replace external Git providers. In fact, they are designed to work with them (GitHub, GitLab, ADO, etc.). They do not provide a built-in Git server in the control plane.

Correct answers: A, B — Use the `include` mapping at the top level of the main `databricks.yml` file to specify the paths to the modular YAML files.; Ensure that each modular YAML file contains a valid Databricks Asset Bundle configuration structure, such as a `resources` block.

• A. Correct. Using the `include` mapping at the top level of the main `databricks.yml` file is the standard method to pull in additional YAML files. This allows the primary configuration file to stay organized while outsourcing specific resource definitions to domain-specific files.
• B. Correct. Each included file must adhere to the Databricks Asset Bundle schema. This means they should contain valid top-level blocks like `resources`, `targets`, or `variables` so that the Databricks CLI can successfully merge them into the total configuration.
• C. Incorrect. The `bundle` block defines the identity and metadata of the bundle as a whole. It should only be defined once in the main `databricks.yml` file, not in the modularized resource files.
• D. Incorrect. Databricks Asset Bundles do not support an `import` keyword within the `resources` block for modularization. The root-level `include` mapping is the correct mechanism.
• E. Incorrect. The Databricks CLI natively handles modular YAML files. There is no need to manually compile or convert them into JSON before running `databricks bundle deploy`.

Correct answer: A — Run the `FSCK REPAIR TABLE` command on the source Bronze Delta table to remove the missing file reference from the transaction log.

• A. `FSCK REPAIR TABLE` (available in Databricks Runtime 9.1 LTS and above) is the official command for repairing a Delta table that has missing or corrupted data files. It updates the transaction log to mark the missing file as removed, restoring the table to a consistent state without discarding any other valid data. This is the most targeted and recommended approach for this specific corruption scenario. (See Databricks documentation on Delta table repair.)
• B. While `RESTORE TABLE` can revert the table to a previous consistent version, it is a broad operation that would roll back the entire table to an earlier point in time. This means any data written after that version—even data unrelated to the deleted file—would be lost. For a missing file, `FSCK REPAIR TABLE` is preferred because it surgically removes the reference to the missing file without affecting other valid data. Use `RESTORE TABLE` for undoing logical mistakes (e.g., bad `MERGE` or `DELETE`), not for physical file corruption.
• C. A full refresh re-processes all data from the source, but it does not repair the source table. The Bronze table still has a missing file and will continue to throw the same corruption error when the pipeline reads from it. The source must first be repaired for the pipeline to function.
• D. Changing the channel only selects a different Databricks Runtime version for the pipeline execution. It has no effect on the underlying storage-level issue of a missing data file. The corruption must be resolved at the Delta table level, independent of the pipeline's runtime.

Correct answer: A — ```sql CREATE TABLE fct_trades ( ... ) CLUSTER BY (stock_ticker, trade_timestamp, trader_id); ```

• A. Correct. According to Databricks documentation, Liquid Clustering (`CLUSTER BY`) is the recommended modern approach for data layout optimization, replacing traditional partitioning and Z-ordering. It excels with high-cardinality columns and simplifies table management. This strategy directly addresses all requirements: - Clustering by `stock_ticker` and `trade_timestamp` optimizes data skipping for the most performance-critical queries. - Including `trader_id` in the clustering keys co-locates data for each trader, significantly speeding up GDPR-related point deletes.
• B. Incorrect. This DDL statement is invalid. According to Databricks official documentation, Liquid Clustering (`CLUSTER BY`) and partitioning (`PARTITIONED BY`) are mutually exclusive and cannot be used on the same table. Liquid Clustering is designed to replace traditional partitioning strategies.
• C. Incorrect. While this is a valid legacy approach, it is not the most scalable or performant modern solution. Liquid Clustering is recommended over partitioning and Z-ordering for new tables as it provides better data layout flexibility and performance without the overhead of managing partition boundaries and running `OPTIMIZE` with `ZORDER`.
• D. Incorrect. Partitioning by `stock_ticker` is an anti-pattern for this use case. `stock_ticker` is a high-cardinality column, which would result in a massive number of small partitions. This leads to the 'small file problem', severely degrading both query and write performance and creating a bottleneck for metadata operations.

Correct answer: B — By removing the need for date-based partitioning altogether, allowing the clustering algorithm to organize data by date fields in a more flexible, adaptive way without creating a nested directory structure.

• A. Incorrect. Liquid Clustering does not store all data in a single massive file; it organizes data into multiple well-sized files based on clustering keys and data distribution. This approach is fundamental to Delta Lake's performance, and a single-file design would not eliminate file listing overhead in the same way.
• B. Correct. Liquid Clustering removes the need for rigid date-based partitioning, allowing the clustering algorithm to organize data by date fields flexibly and adaptively without creating a deep nested directory structure. This decoupling of physical layout from partitioning eliminates the file listing overhead caused by high-cardinality partitions.
• C. Incorrect. Liquid Clustering does not rely on a metadata cache to avoid directory traversal; instead, it fundamentally changes the data layout to prevent deep directory structures from forming. While caching mechanisms exist in Databricks, they are not the primary method Liquid Clustering uses to solve file listing performance issues.
• D. Incorrect. Liquid Clustering does not replace the file system with a key-value store; it operates on the same underlying object storage (e.g., S3, ADLS, GCS) as standard Delta tables. It optimizes file layout within the existing file system rather than introducing a separate atomic store for partition-to-file mappings.

Correct answer: B — Use generated columns to extract key-value pairs like `page_url` from the `payload` column and then Z-ORDER by `user_id` and the generated column.

• A. Incorrect. Creating separate tables for each event type and combining them with a `UNION ALL` view would require significant changes to the ingestion pipeline, violating the requirement to avoid altering it. This approach also adds complexity in managing multiple tables and views, and `UNION ALL` operations can be less performant than querying a single, well-optimized table.
• B. Correct. Generated columns allow extraction of nested JSON fields like `page_url` into top-level columns without changing the ingestion pipeline, enabling Delta Lake to collect statistics for data skipping. Z-ORDERing by `user_id` and the generated column co-locates related data, dramatically improving query performance for filters on these attributes.
• C. Incorrect. Implementing a UDF to parse JSON at query time prevents the Catalyst optimizer from performing predicate pushdown and other optimizations, leading to poor performance. While Delta Lake can query nested data, using generated columns is a far more efficient method for frequently filtered nested fields.
• D. Incorrect. Adding `user_id` as a partition key would create a large number of small partitions due to its high cardinality, causing the small file problem and degrading query performance. Partitioning is best suited for low-to-moderate cardinality columns, not high-cardinality identifiers like `user_id`.