1.4 Outline and use best practice considerations relating to data integrity structures.
1.A company ingests customer data from multiple sources into a staging table. They want to create a final `DIM_CUSTOMER` table where `CUSTOMER_NK` (natural key) is unique. Since Snowflake doesn't enforce uniqueness, which combination of Snowflake features should be used to identify and handle duplicate `CUSTOMER_NK` values during the ELT process before inserting into the final table?(Select 2)
- A.Define a UNIQUE constraint on `CUSTOMER_NK` in the `DIM_CUSTOMER` table to automatically reject duplicates.
- B.Use a `MERGE` statement with a `WHEN NOT MATCHED` clause to insert new customers.
- C.Use the `QUALIFY ROW_NUMBER() OVER (PARTITION BY CUSTOMER_NK ORDER BY load_timestamp DESC) = 1` clause in a SELECT statement from the staging table.
- D.Use the `GROUP BY CUSTOMER_NK HAVING COUNT(*) > 1` clause to identify duplicates before the main load.
- E.Rely on the `COPY INTO` command's `VALIDATION_MODE` to report uniqueness violations.
Show answer & explanation
Correct answers: B, C — Use a `MERGE` statement with a `WHEN NOT MATCHED` clause to insert new customers.; Use the `QUALIFY ROW_NUMBER() OVER (PARTITION BY CUSTOMER_NK ORDER BY load_timestamp DESC) = 1` clause in a SELECT statement from the staging table.
- A. Incorrect. While Snowflake allows the definition of UNIQUE constraints, it does not enforce them (with the exception of NOT NULL). These constraints serve primarily as metadata for query optimization and data modeling tools and will not prevent duplicate rows from being inserted.
- B. Correct. A `MERGE` statement is ideal for synchronizing two tables. Its `WHEN NOT MATCHED` clause allows for the insertion of rows from a source that do not exist in the target based on a specified key. This effectively prevents the creation of duplicate records that are already present in the final dimension table, making it a key part of an idempotent loading process.
- C. Correct. This is a highly efficient and common Snowflake pattern to de-duplicate data. The `ROW_NUMBER()` window function assigns a rank to each row within a partition of the `CUSTOMER_NK`. By using the `QUALIFY` clause to filter for rows where the rank is 1, you can select a single, consistent record (e.g., the most recent one) for each natural key, thereby handling duplicates within the staging table itself before the final load.
- D. Incorrect. This clause is effective for *identifying* which natural keys have duplicate entries in the staging table. However, it does not *handle* them, as it only returns the duplicated keys themselves, not a clean set of unique records to be inserted. Additional logic would be required to select the desired record from among the duplicates.
- E. Incorrect. The `COPY INTO` command's `VALIDATION_MODE` is used to validate data files for parsing errors, data type mismatches, or structural issues before loading. It does not check for uniqueness against data already existing in the target table.