Free Cloudera Data Engineer (CDP-3002) Sample Questions

Correct answers: A, C, E, F — Reduce the watermark delay; Use RocksDB for state storage; Enable state store compression; Increase executor memory

• A. Correct. Reducing the watermark delay causes Spark to drop late data more aggressively, allowing older state to be evicted sooner. This directly reduces the amount of state maintained, lowering memory pressure. However, it must be balanced against the tolerance for late-arriving data.
• B. Incorrect. Increasing spark.sql.shuffle.partitions changes the parallelism for shuffle operations but does not directly reduce the total amount of state kept by stateful aggregations. It is not an effective measure for reducing state size or memory usage.
• C. Correct. Using RocksDB for state storage offloads state management from the JVM heap to disk-based storage, significantly reducing memory pressure. RocksDB is designed for large stateful workloads and improves stability by handling state more efficiently.
• D. Incorrect. Decreasing the trigger interval increases the frequency of micro-batches, which leads to more frequent state updates and higher overhead. It does not reduce state size and can actually increase memory usage due to more processing cycles.
• E. Correct. Enabling state store compression reduces the on-disk and in-memory footprint of state data. This directly lowers memory usage and helps manage large state, improving stability without altering the logical state structure.
• F. Correct. Increasing executor memory provides more heap and off-heap resources to the streaming query. While it does not reduce the size of state, it helps prevent out-of-memory failures and improves stability when state is legitimately large. It is a valid approach to improve stability.

Correct answer: B — spark.dynamicAllocation.executorIdleTimeout

• A. Incorrect. spark.dynamicAllocation.minExecutors sets the minimum number of executors to keep alive, but it does not control the duration idle executors are retained before being removed. Increasing it may retain more executors overall but is not the timeout setting for idle removal.
• B. Correct. spark.dynamicAllocation.executorIdleTimeout specifies the time in seconds an executor can be idle before it is removed. Increasing this value keeps idle executors alive longer, reducing the performance penalty of recreating executors when new tasks arrive.
• C. Incorrect. spark.dynamicAllocation.schedulerBacklogTimeout controls how long the scheduler waits with pending tasks before requesting new executors. It affects scaling up under load, not the idle timeout for existing executors.
• D. Incorrect. spark.dynamicAllocation.maxExecutors defines the maximum number of executors Spark can allocate. It limits scale-out capacity but does not affect how long idle executors are retained before removal.

Correct answer: B — 10 executors, 4 cores each, 27GB memory each

• A. Incorrect. While this configuration uses 5 executors (one per node) with 9 cores each, it matches the total usable cores (45) but does not maximize parallelism. Fewer executors limit the number of concurrent tasks. Additionally, large executors can lead to longer garbage collection times and slower failure recovery.
• B. Correct. This configuration uses 10 executors (2 per node) with 4 cores each, leaving 1 core per node for system processes, respecting the 10-core limit. With 40 executor cores, it allows more concurrent tasks than fewer large executors, maximizing parallelism. The 27GB memory per executor is reasonable after accounting for OS and YARN overhead on 64GB nodes.
• C. Incorrect. This configuration assigns 12 cores per executor, which exceeds the 10 cores available per node, and would require 48 executor cores total, exceeding the 45 usable cores after reserving one per node. The 60GB memory per executor is also too aggressive for 64GB nodes, leaving insufficient room for OS and daemon processes.
• D. Incorrect. A single executor with 49 cores is not possible because the cluster only has 45 usable cores after reserving one per node. This configuration would also destroy parallelism by concentrating all work into one JVM, which is the opposite of the goal of maximum parallelism.

Correct answer: C — Use a Kubernetes Secret of type docker-registry and reference it via spark.kubernetes.container.image.pullSecrets

• A. Incorrect. While you can configure an imagePullSecret on a pod or service account, Spark on Kubernetes does not automatically use these. The recommended approach is to set the Spark configuration property spark.kubernetes.container.image.pullSecrets to reference a secret.
• B. Incorrect. There is no Spark configuration property named spark.kubernetes.registry.credentials. This is not a valid way to authenticate with a private registry in Spark on Kubernetes.
• C. Correct. This is the recommended approach. Create a Kubernetes Secret of type docker-registry and set the property spark.kubernetes.container.image.pullSecrets to the secret name. Spark will use that secret when pulling images from the private registry.
• D. Incorrect. Adding a CA certificate via hostPath does not provide authentication credentials. It only helps with TLS certificate trust, not with username/password or token-based registry authentication.

Correct answer: B — spark.decommission.enabled

• A. Incorrect. This property is used for dynamic allocation shuffle tracking in general Spark environments, but it is not the primary property for enabling shuffle tracking during decommissioning on Kubernetes. It does not directly address executor decommissioning.
• B. Correct. Enabling spark.decommission.enabled allows Spark to safely decommission executors, including preserving and tracking shuffle files so that they can be reused by other executors. This is the standard configuration for executor decommissioning support in Spark, including on Kubernetes.
• C. Incorrect. While Spark has some Kubernetes-specific shuffle behaviors, this exact property is not a standard Spark configuration key. The correct approach for shuffle preservation on Kubernetes involves other settings, and this property is not used to enable shuffle tracking.
• D. Incorrect. This property controls the use of an older shuffle fetch protocol and is unrelated to shuffle tracking or executor decommissioning. It does not enable Spark to preserve shuffle data when executors are removed.

Correct answer: B — spark.kubernetes.executor.jvm.options

• A. Incorrect. This is the standard Spark setting for passing JVM options to executors in standalone or YARN mode, but it is not the Kubernetes-specific configuration key. On Kubernetes, Spark uses dedicated Kubernetes-prefixed options for executor JVM settings.
• B. Correct. spark.kubernetes.executor.jvm.options is the Kubernetes-specific Spark configuration property used to supply JVM options to executor pods. It is the appropriate choice when configuring executor JVM options on Spark running over Kubernetes.
• C. Incorrect. This setting applies JVM options to the driver, not the executors. It does not configure executor JVM behavior on Kubernetes.
• D. Incorrect. This is not a valid Spark configuration property. The correct Kubernetes-specific property is spark.kubernetes.executor.jvm.options.

Correct answer: A — Create a Kubernetes Secret and mount it using spark.kubernetes.executor.secrets.<keytab>

• A. Correct. Creating a Kubernetes Secret is the recommended secure method for storing sensitive data like keytabs. Spark supports mounting secrets into executor pods using spark.kubernetes.executor.secrets.<name>, providing secure access to the keytab file.
• B. Incorrect. Base64 encoding is not encryption and does not secure the keytab. Passing it as an environment variable is insecure because environment variables can be exposed in logs, process listings, or pod specs.
• C. Incorrect. ConfigMaps are intended for non-sensitive configuration data, not credentials. A keytab contains secret authentication material and should be stored in a Kubernetes Secret, not a ConfigMap.
• D. Incorrect. spark.kerberos.keytab is used to specify the path to a keytab file on the local filesystem of the Spark driver or executor, not to retrieve from HDFS. It does not handle secure distribution of the keytab to Kubernetes pods.

Correct answer: A — BashOperator

• A. Correct. The BashOperator is specifically designed to execute bash commands or scripts in an Airflow DAG, allowing you to run shell commands directly on the worker.
• B. Incorrect. The PythonOperator is used to execute Python functions, not bash commands. It is for running Python code within a task.
• C. Incorrect. The SSHOperator is used to execute commands on a remote machine via SSH, not for running local bash commands. While it can run shell commands remotely, it is not the primary operator for local bash execution.
• D. Incorrect. The SimpleHttpOperator is used to make HTTP requests to web services, not to execute bash commands.

Correct answer: A — task_a >> task_b

• A. Correct. The bitshift operator (>>) is the standard and recommended way to define task dependencies in Airflow. Using task_a >> task_b means task B runs after task A completes successfully.
• B. Incorrect. task_b.set_downstream(task_a) reverses the dependency; it would mean task A runs after task B, which is the opposite of the desired order.
• C. Incorrect. task_a.set_upstream(task_b) also reverses the dependency; it would mean task A depends on task B, so task B runs before task A, not after.
• D. Incorrect. DAG.set_dependency() is not a valid Airflow method. Dependencies are defined directly between tasks using bitshift operators (>>, <<) or set_downstream/set_upstream methods on task objects.

Correct answer: A — It waits for an external condition before proceeding.

• A. Correct. A Sensor is designed to wait until an external condition (e.g., file arrival, database record, API response, partition existence) becomes true before allowing downstream tasks to continue. It blocks execution until the condition is met.
• B. Incorrect. Sensors do not measure execution time; task duration is captured by Airflow logs, metrics, and monitoring tools. Sensors focus on waiting for conditions, not timing tasks.
• C. Incorrect. Sending notifications on task failure is handled by callbacks, alerts, or integrations like email or Slack operators, not by Sensors. Sensors are for waiting on conditions.
• D. Incorrect. Controlling concurrent tasks is managed by Airflow pools, concurrency limits, and executor settings, not by Sensors. Sensors do not manage scheduler concurrency.

Correct answer: B — GreatExpectationsOperator

• A. Incorrect. SQLExecuteQueryOperator is used to execute SQL queries against a database, but it does not natively perform schema validation. While it can be part of a validation workflow with custom SQL checks, that is not its primary purpose or a built-in feature.
• B. Correct. The GreatExpectationsOperator is specifically designed for data validation and can check datasets against predefined schemas, expectations, or quality rules. It is the most appropriate choice for schema validation in Airflow.
• C. Incorrect. PythonOperator can execute arbitrary Python code, so schema validation could be implemented manually inside a Python function. However, it is not a dedicated operator for schema validation and does not provide built-in predefined schema checking.
• D. Incorrect. BashOperator runs shell commands and scripts, which could invoke external validation tools, but it does not itself validate data against a schema. Any schema checking would have to be done by the script it launches.

Correct answer: A — Set `retries` and `retry_delay` on the check task

• A. Setting `retries` and `retry_delay` on the `SQLColumnCheckOperator` task allows Airflow to automatically retry the task a specified number of times with a delay between attempts. This is the standard way to handle transient failures such as late-arriving source data, as it directly retries the failed check until the data is available.
• B. A `TimeDeltaSensor` waits for a fixed duration to pass, not for the data to become available. It does not retry the failed check based on data readiness and is not suitable for handling late-arriving data.
• C. A `ShortCircuitOperator` conditionally skips downstream tasks if a condition is false. It does not provide retry behavior for a failing quality check and does not address the need to retry the check until data arrives.
• D. `TriggerDagRunOperator` is used to trigger a separate DAG run, not to retry the current task. This adds orchestration complexity without solving the problem of delaying and retrying the quality check within the same pipeline.

Correct answer: A — Identify fields that support Jinja templating

• A. Correct. The `template_fields` attribute in a custom Airflow operator specifies which fields of the operator should be processed as Jinja templates at runtime, allowing dynamic values like SQL, table names, or dates to be injected when the task executes.
• B. Incorrect. Retry behavior and delays are controlled by parameters such as `retries` and `retry_delay`, not by `template_fields`.
• C. Incorrect. The task's display color in the UI is set using other operator properties or UI settings, not by `template_fields`.
• D. Incorrect. The execution schedule interval is defined at the DAG level via the `schedule_interval` parameter, not by an operator's `template_fields`.

Correct answer: A — It controls whether the DAG creates runs for past intervals that were missed since the start_date.

• A. Correct. The `catchup` parameter, when set to `False`, prevents Airflow from creating DAG runs for past intervals between the `start_date` and the current time. This avoids backfilling missed runs.
• B. Incorrect. The `catchup` parameter does not control future schedule intervals. It only affects past intervals relative to the `start_date`. Airflow continues to schedule future runs normally.
• C. Incorrect. The `catchup` parameter does not limit the number of intervals during backfill. It simply enables or disables backfilling entirely, not cap the number of intervals.
• D. Incorrect. The `catchup` parameter is unrelated to retrying failed task instances. Retries are controlled by the `retries` parameter in tasks, not by the DAG's `catchup` setting.

Correct answer: C — Return the value from PythonOperator and access it in BashOperator using XCom and Jinja.

• A. Writing to a shared file can work but introduces unnecessary complexity, potential race conditions, and file management. It is not the idiomatic Airflow-native approach for passing small values between tasks.
• B. Airflow Variables are intended for static configuration, not for dynamic task-to-task communication. Using Variables for runtime data is not recommended.
• C. XCom (cross-communication) is the built-in mechanism for sharing small amounts of data between tasks. The PythonOperator can return the value, and the BashOperator can access it via XCom and Jinja templating (e.g., `{{ ti.xcom_pull(task_ids='python_task') }}`). This is the recommended pattern.
• D. Using a temporary database table is overkill for a small string and adds unnecessary dependencies and complexity. Airflow provides XCom specifically for lightweight task-to-task data exchange.

Correct answer: B — Configure execution_timeout with a timedelta value in the task's parameters.

• A. Incorrect. The sla parameter is used for monitoring and alerting when a task exceeds expected completion time, but it does not enforce a hard timeout or fail the task automatically.
• B. Correct. execution_timeout is a task-level parameter that accepts a timedelta value. If the task runs longer than this duration, Airflow automatically marks it as failed, directly addressing hanging tasks.
• C. Incorrect. dagrun_timeout is a DAG-level parameter that limits the total runtime of a DAG run, not individual tasks. It fails the entire DAG run if exceeded, but does not enforce per-task limits.
• D. Incorrect. retry_delay specifies the wait time between task retries after a failure, but it does not limit the runtime of a task or cause it to fail due to excessive duration.

Correct answer: A — Persist the last processed record ID via XCom; on retry, the operator reads that ID to resume extraction.

• A. Correct. Persisting a checkpoint (e.g., the last processed record ID) allows the operator to resume extraction from the correct point if the task is retried. While XCom is not a durable store across DAG runs (it persists only for the duration of the DAG run), it is the only option that explicitly implements idempotency by tracking progress. In practice, a database or external storage is recommended, but among the given choices, this is the best approach.
• B. Incorrect. A full extract is not an incremental extraction strategy. It is inefficient for large datasets and relies on the target system's deduplication, which may not be deterministic or robust. This approach does not make the extraction process itself idempotent.
• C. Incorrect. Exponential backoff manages retry timing but does not prevent duplicate data. Relying on the source system for exactly-once delivery is not a reliable assumption and does not ensure idempotency in the extraction job.
• D. Incorrect. Checking for existing data before insertion prevents duplicate records in the target, but it does not make the extraction process idempotent. The extraction may still re-extract the same data, and the deduplication check is inefficient for large volumes. It does not provide a durable checkpoint for resuming after failures.

Correct answer: B — SparkSubmitOperator

• A. Incorrect. PythonOperator executes Python functions within Airflow but does not natively submit or manage Spark jobs on a cluster. While it could be used to prepare parameters, it lacks the built-in capabilities for Spark job submission and cluster interaction required for this task.
• B. Correct. SparkSubmitOperator is specifically designed to submit Spark applications to a cluster. It seamlessly integrates with Airflow context, allowing you to pass start and end timestamps as parameters, making it the most appropriate operator for incremental extraction from a Hive table using Spark.
• C. Incorrect. BashOperator runs shell commands, which could indirectly invoke spark-submit, but this approach adds unnecessary complexity and lacks native support for passing Airflow context parameters directly to the Spark job. The dedicated SparkSubmitOperator is more appropriate.
• D. Incorrect. HiveOperator is used to execute Hive SQL queries, not Spark jobs. Even though the data source is a Hive table, the requirement specifies running an Apache Spark job on a cluster, which is not achievable with this operator.

Correct answer: C — Retrieve last processed timestamp from state table, query API using that timestamp as 'modified_after', after loading update table with max event timestamp.

• A. Incorrect. XCom is not durable across retries or DAG runs; it is ephemeral and tied to a specific task instance, so it won't persist on failure. A sensor does not solve the state management problem required for exactly-once incremental extraction.
• B. Incorrect. Pulling all events without a timestamp filter is inefficient and does not guarantee exactly-once processing. Deduplication in the target may avoid duplicates but does not implement proper incremental extraction from the source system.
• C. Correct. A durable state table (e.g., in a database) persists across retries and DAG runs. Using the stored timestamp for the 'modified_after' parameter and updating it only after a successful load ensures incremental, exactly-once extraction.
• D. Incorrect. Using execution_date minus a buffer does not guarantee exactly-once processing; it may miss events or reprocess them, especially on retries. Overwriting partitions can lead to duplicates or gaps.

Correct answer: B — Jobs tab

• A. Incorrect. The Stages tab shows stage-level details such as task distribution, shuffle read/write, and stage metrics, but it does not display the DAG visualization of RDD dependencies for the entire job.
• B. Correct. The Jobs tab in the Spark UI provides a DAG visualization for each job, illustrating the RDD dependencies and the sequence of operations. This is the place to inspect the logical flow of a job.
• C. Incorrect. The SQL tab is used for Spark SQL queries and their execution details, such as query plans and SQL metrics. It is not the main tab for viewing a job-level DAG of RDD dependencies and operations.
• D. Incorrect. The Storage tab displays persisted RDDs, cached DataFrames, and memory/disk usage. It does not show the DAG visualization for a job.

Correct answer: B — The amount of data read from the shuffle by the stage's tasks

• A. Incorrect. This describes shuffle write, not shuffle read. The 'Shuffle Write Size/Records' column represents data written to disk by the stage for shuffle, whereas 'Shuffle Read' tracks data fetched during shuffle processing.
• B. Correct. The 'Shuffle Read Size/Records' column shows the amount of data and number of records read from shuffle outputs by the stage's tasks during a shuffle operation. This metric reflects the data fetched from other executors for intermediate data exchange between stages.
• C. Incorrect. The total input data read from sources like HDFS or S3 is represented by the 'Input Size/Records' column. 'Shuffle Read Size/Records' specifically refers to intermediate data exchanged between stages, not source input.
• D. Incorrect. Broadcast variable size is tracked separately and is not part of shuffle metrics. Broadcast data is sent to executors for joins or lookups, but its size is not reflected in the 'Shuffle Read Size/Records' column.

Correct answer: B — GC Time

• A. Incorrect. Total Duration refers to the overall runtime associated with an executor or task context, not the time specifically spent in garbage collection. It is a broad metric and does not isolate GC overhead.
• B. Correct. GC Time is the metric in the Spark UI Executors tab that explicitly tracks the cumulative time spent by the executor on garbage collection. It is critical for identifying performance bottlenecks related to JVM memory pressure and tuning needs.
• C. Incorrect. Shuffle Read Write Time measures the time spent on shuffle operations (reading from and writing to shuffle data), which is unrelated to garbage collection activity.
• D. Incorrect. Serialization Time measures the time taken to serialize and deserialize data. While it can affect performance, it does not represent garbage collection time.

Correct answers: B, D — Stages tab; Executors tab

• A. The Jobs tab provides a high-level overview of job status and duration but lacks the per-task granularity needed to identify skew. Skew is best detected by examining stage and task metrics, not the job summary.
• B. The Stages tab is essential as it breaks down jobs into stages and tasks, showing per-task duration, input/output sizes, and skew metrics like task duration variance. It is the primary tool for detecting skewed task execution.
• C. The Storage tab is used for managing cached/persisted RDDs and DataFrames, not for monitoring task execution or skew.
• D. The Executors tab provides per-executor metrics such as task time, input/output, and GC time, which can reveal workload imbalance across executors. This is valuable for identifying skew, especially when certain executors are overloaded.
• E. The Environment tab shows Spark configuration and runtime settings, unrelated to task execution or skew diagnostics.
• F. The SQL tab focuses on query execution plans and SQL-specific metrics, not general task execution or skew. It is not essential for the core task monitoring purpose.

Correct answer: B — The operator is part of whole-stage code generation.

• A. Incorrect. The `*` prefix does not indicate columnar processing. Columnar processing is related to Spark's internal execution engine (e.g., Tungsten) but is not denoted by this symbol.
• B. Correct. The `*` prefix marks operators that are included in a whole-stage code generation pipeline. This means Spark has optimized the execution by fusing multiple operators into a single Java bytecode function for better performance.
• C. Incorrect. While Spark does optimize operators, the `*` prefix specifically refers to whole-stage code generation, not a generic replacement by an optimized version.
• D. Incorrect. Broadcast operations are typically denoted by other indicators (e.g., BroadcastHashJoin or BroadcastExchange) and not by the `*` prefix.

Correct answer: D — Per operator (node)

• A. Incorrect. Metrics like number of output rows are not aggregated at the task level in an explain plan. Task-level granularity is too fine; these details are available in runtime UIs or logs, not in the logical/physical plan output.
• B. Incorrect. While stages are a Spark execution unit, explain plan metrics are reported at a finer granularity per operator, not summarized per stage. Stage-level aggregate metrics appear in execution monitoring views, not the operator-level explain plan.
• C. Incorrect. Executors are runtime JVM processes that execute tasks; metrics per executor relate to cluster resource monitoring, not the operator-level metrics shown in an explain plan. The explain plan focuses on the logical and physical plan operators.
• D. Correct. In an explain plan with metrics, the number of output rows and similar metrics are reported per operator (node). Each node in the plan represents a specific operation (e.g., filter, join), and metrics are collected at that granularity, aiding in performance bottleneck identification.

Correct answer: C — /*+ MERGE */

• A. Incorrect. The BROADCAST hint tells Spark to broadcast one side of the join to avoid a shuffle, typically resulting in a broadcast hash join. It does not instruct Spark to use a sort-merge join.
• B. Incorrect. The SHUFFLE_HASH hint prefers a shuffle hash join, where both sides are shuffled and hashed. This is different from a sort-merge join, which sorts both sides before merging.
• C. Correct. The MERGE hint explicitly instructs Spark to use a sort-merge join, which sorts both datasets on the join key and then merges them. This join strategy is often used for large datasets.
• D. Incorrect. The SHUFFLE_REPLICATE_NL hint forces a shuffled replicated nested-loop join, generally used for cross joins or certain non-equi joins. It does not select sort-merge join behavior.

Correct answer: A — It requires multiple passes over the data, increasing I/O.

• A. Schema inference in Spark typically scans the data (or a sample) to determine column types. While it does not necessarily require multiple full passes, the additional read overhead increases I/O and can degrade performance, especially with large datasets. This is a recognized drawback compared to providing an explicit schema.
• B. Schema inference does not inherently cause a full shuffle across the cluster. Shuffles are triggered by operations like joins, aggregations, or repartitioning, not by schema inference itself.
• C. Schema inference does not force conversion to Parquet. Spark can infer schemas from various formats natively (e.g., CSV, JSON, Avro) without requiring an intermediate Parquet conversion.
• D. Schema inference is not limited by an absolute file-size threshold like 10 GB. While reading very large files may increase overhead, there is no hard limit that prevents handling larger files.

Correct answer: B — spark.read.option("inferSchema", "false").csv(path)

• A. Incorrect. Providing an explicit schema with `.schema(mySchema)` does override schema inference, but it is not the standard way to disable inference via reader options. This approach uses a predefined schema rather than disabling inference through the `inferSchema` option.
• B. Correct. Setting the option `inferSchema` to `false` explicitly tells Spark not to infer data types from the CSV contents, forcing it to use default string-based reading. This is the standard method to read a CSV without schema inference.
• C. Incorrect. The `samplingRatio` option controls the fraction of data sampled for schema inference, but setting it to `0` does not disable inference entirely. Spark may still attempt inference with a minimal sample or fall back to defaults. The correct way is to set `inferSchema` to `false`.
• D. Incorrect. The `schema` parameter is not a valid option passed via `.option()` for CSV reading. Spark expects the schema to be set with `.schema(mySchema)`, not through this option. This line would not compile or would be ignored.

Correct answers: A, D — Provide an explicit schema for all JSON reads.; Disable schema inference for production jobs.

• A. Providing an explicit schema avoids the overhead of schema inference, improves performance, and ensures consistent data types. This is a key best practice for production workloads.
• B. Incorrect. The 'inferSchema' option is primarily for CSV, not JSON. JSON schema inference occurs automatically if no schema is provided, but using inference is not recommended due to performance overhead and potential type inconsistencies.
• C. Incorrect. FAILFAST mode is used to fail immediately on malformed records, but it does not directly help identify schema mismatches or improve schema inference performance. It is more about error handling.
• D. Correct. Disabling schema inference in production jobs by providing a known schema eliminates extra data scanning, improves performance, and reduces the risk of schema inconsistencies across runs.
• E. Incorrect. Using a sampling ratio trades accuracy for speed and can lead to missing fields or incorrect column types. It is not a best practice for production; providing an explicit schema is preferred.

Correct answer: B — Run MSCK REPAIR TABLE to update the metastore with new partitions.

• A. Incorrect. Partition addition is supported for all partitioned tables, not just bucketed tables. Bucketing is a separate optimization concept and does not restrict partition operations.
• B. Correct. MSCK REPAIR TABLE is a Hive command used to discover and register new partitions in the metastore, especially for external tables where partitions are added via the filesystem. It synchronizes the partition metadata with the directory structure.
• C. Incorrect. Empty partition directories are still recognized as partitions by the metastore when MSCK REPAIR TABLE is run. Partition discovery depends on the directory structure, not on the presence of data files.
• D. Incorrect. File format does not affect partition recognition. Partitions are recognized based on the directory path and metadata registration; file format only impacts read capabilities.

Correct answer: B — Repartition the US data by a granular column, overwriting the partition dynamically.

• A. Incorrect. Increasing the bucket count helps distribute data within a partition but does not reduce the size of an oversized partition. Bucket count alone does not address partition-level skew; the large partition remains large, and performance issues persist.
• B. Correct. Repartitioning the US data by a more granular column (e.g., state or city) creates smaller, more evenly sized partitions. Using dynamic overwrite allows updating only the affected partition without rewriting the entire table, directly reducing skew.
• C. Incorrect. Broadcast join is used for small tables to avoid shuffling, but it is not applicable here because the large partition itself is not a join partner; the skew is in table storage. Broadcasting a large partition would cause memory issues.
• D. Incorrect. Adaptive skew join handling is a runtime optimization for join operations, not a storage-layout solution. It does not rebalance partition sizes in the table; it only mitigates skew during query execution and does not fix the underlying partition imbalance.

Correct answer: B — Rewrite the filter to use a direct date range without a function.

• A. Incorrect. Repartitioning by year does not solve pruning issues when a function is applied to the partition column. Partition pruning works only when the filter directly matches the partition column without transformation.
• B. Correct. Rewriting the filter to use a direct date range (e.g., `date_column >= '2023-01-01' AND date_column < '2024-01-01'`) allows Hive/Spark to perform partition pruning and use metadata efficiently. This makes the predicate sargable and avoids full table scans.
• C. Incorrect. Hive and Spark SQL do not rely on traditional indexes. Even if indexes were available, they would not circumvent the function-based filter issue. The proper solution is to rewrite the query to avoid functions on the partition column.
• D. Incorrect. Increasing the number of buckets improves parallelism for joins and aggregations, but does not address filtering inefficiencies caused by a function on the date column. Bucketing is not related to partition pruning.

Correct answer: B — Manifest list

• A. A manifest file stores metadata about data files (e.g., Parquet files) and their statistics, but it does not list other manifest files; it is a leaf-level metadata file.
• B. A manifest list is the correct metadata file that enumerates all manifest files belonging to a specific snapshot, acting as an index for the manifests in that snapshot.
• C. Snapshot log is not a standard Iceberg metadata file type; snapshot history is maintained within the table metadata file, but it does not directly list manifest files.
• D. The table metadata file contains high-level table information such as schema, partition specs, and snapshot references, but it does not directly list the manifest files for a given snapshot; that role belongs to the manifest list.

Correct answer: B — It maintains the sequence of valid snapshots with timestamps

• A. Incorrect. The snapshot log does not record partitions with data. Partition information is stored in manifest files and table metadata, not in the snapshot log.
• B. Correct. The snapshot log maintains the sequence of valid snapshots and their timestamps, enabling time travel and historical metadata tracking.
• C. Incorrect. The snapshot log does not store a full history of data file manipulations; data file changes are tracked indirectly through snapshots, manifests, and manifest lists.
• D. Incorrect. The snapshot log's primary role is to record snapshot metadata and ordering, not to track the lineage of manifest files, which is managed separately in the metadata.

Correct answer: A — By tracking schema versions and letting readers use the version they understand

• A. Correct. Iceberg maintains a history of schema versions, using stable field IDs to allow readers to resolve fields regardless of position or name. This enables readers to use the schema version they understand, ensuring backward compatibility and preventing disruption from schema changes.
• B. Incorrect. Iceberg does not force backward-incompatible changes; it supports schema evolution without requiring full data rewrites. This is a key benefit of Iceberg.
• C. Incorrect. Iceberg explicitly supports schema evolution (adding, dropping, renaming, reordering columns) through metadata and field IDs, not a fixed schema.
• D. Incorrect. Readers do not need to drop and recreate tables; Iceberg's metadata layer allows schema changes without disrupting existing readers.