Subdomain 1.2: Building and Testing an ETL pipeline with Lakeflow Spark Declarative Pipelines, SQL, and Apache Spark on the Databricks Platform
1.A team is deciding between hand-writing a Spark Structured Streaming job with `foreachBatch` and building the same ingestion-to-gold flow as a Lakeflow Spark Declarative Pipeline. The pipeline needs automatic dependency resolution across bronze, silver, and gold tables and built-in data quality tracking. Which approach best fits, and why?
- A.Use a Lakeflow Spark Declarative Pipeline, since it declaratively infers table dependencies and orchestration order and integrates expectations for quality tracking without extra orchestration code
- B.Use hand-written Structured Streaming with `foreachBatch`, since it automatically infers dependency order across arbitrarily named DataFrames without any additional configuration
- C.Use hand-written Structured Streaming with `foreachBatch`, since it natively records data quality metrics into a queryable event log without any custom instrumentation
- D.Use a Lakeflow Spark Declarative Pipeline, but only because it eliminates the need for checkpoints entirely, a limitation Structured Streaming jobs have no way to avoid on their own
Show answer & explanation
Correct answer: A — Use a Lakeflow Spark Declarative Pipeline, since it declaratively infers table dependencies and orchestration order and integrates expectations for quality tracking without extra orchestration code
- A. Correct. Lakeflow Spark Declarative Pipelines infer the dependency graph between declared tables and orchestrate the run order automatically, and expectations provide built-in, queryable data quality tracking, matching both stated requirements directly.
- B. Incorrect. Hand-written Structured Streaming jobs require the engineer to manually sequence and orchestrate bronze, silver, and gold writes; there is no automatic dependency inference between separate `foreachBatch` streams.
- C. Incorrect. Structured Streaming has no built-in expectations mechanism or event log for data quality; any such tracking would need to be custom-instrumented by the engineer, unlike the pipeline framework's native support.
- D. Incorrect. Lakeflow Spark Declarative Pipelines still rely on checkpoints internally to track streaming progress; the framework manages checkpoint locations for the engineer rather than eliminating the underlying need for them.