What you will be able to do
- Explain what the cloudFiles source does and why it beats the plain Spark file source for ingestion
- Describe how Auto Loader's checkpoint delivers exactly-once processing, and what Lakeflow pipelines manage for you
- Choose between a streaming table, a materialized view and a view for each layer of an ingestion pipeline
- Predict how a streaming table behaves when its query changes, when its stream is unbounded, and when the source schema drifts
Key concept
Streaming table fed by Auto Loader — A Unity Catalog managed Delta table that a pipeline appends to incrementally. Auto Loader's checkpoint records which files it has already ingested, so each new record is processed exactly once and never re-read.
1.Auto Loader: the cloudFiles streaming source
A production streaming pipeline on Databricks usually starts with files landing in cloud storage. Auto Loader is the ingestion engine for that job. It provides a Structured Streaming source named cloudFiles. You point it at an input directory and it processes new files as they arrive, and it can also process the files already in that directory. It reads from Amazon S3 (s3://), ADLS (abfss://), GCS (gs://), Unity Catalog volumes (/Volumes/) and the deprecated WASB driver (wasbs://). Supported formats are JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT and BINARYFILE, including pre-compressed files. You can use Auto Loader from both Python and SQL inside Lakeflow pipelines.
Plain Apache Spark can also read files incrementally with spark.readStream.format(fileFormat).load(directory). So why does Databricks recommend Auto Loader whenever you stream from cloud object storage? The documented reasons are scale, discovery cost and schema handling:
| Dimension | Auto Loader behaviour |
|---|---|
| Scalability | Discovers billions of files efficiently; backfills can run asynchronously |
| Performance | Discovery cost scales with the number of files ingested, not the number of directories they land in |
| Schema | Detects schema drift, notifies you of changes, and rescues data that would otherwise be lost |
| Cost | Uses native cloud APIs to list files; file notification mode can avoid directory listing altogether |
Checkpoint 1 of 5· Check yourself
A team ingests JSON files from an S3 landing path using spark.readStream.format("json"). Discovery slows down as the number of date-partitioned directories grows. Which Auto Loader property addresses this directly?
The documentation says Auto Loader's discovery cost tracks the number of files ingested, not the number of directories they land in. It makes no ordering guarantee, and it supports JSON.
“The cost of discovering files with Auto Loader scales with the number of files that are being ingested instead of the number of directories”Source: docs.databricks.com
Sources1
2.Exactly-once through the checkpoint, and what pipelines manage for you
Every file Auto Loader discovers has its metadata written to a RocksDB key-value store in the stream's checkpoint location. This record of seen files is what stops a file being ingested twice. After a failure, Auto Loader reads the checkpoint, resumes where it left off, and keeps its exactly-once guarantee when writing to Delta Lake. You don't have to manage any of that state yourself.
If you run Auto Loader in plain Structured Streaming, you wire up the checkpoint yourself. You set cloudFiles.schemaLocation to turn on schema inference and evolution, and checkpointLocation on the writer. If you have more than one source location, each Auto Loader ingestion workload needs its own streaming checkpoint.
(spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "parquet")
# The schema location directory keeps track of your data schema over time
.option("cloudFiles.schemaLocation", "<path-to-schema>")
.load("<path-to-source-data>")
.writeStream
.option("checkpointLocation", "<path-to-checkpoint>")
.start("<path-to-target>")
)Inside a Lakeflow pipeline, both of those settings go away. Databricks recommends Auto Loader in Lakeflow pipelines for incremental ingestion. The pipeline manages the schema location and checkpoint for you. It also orchestrates the flows: it works out the dependencies between your datasets and runs them in order, with as much parallelism as possible. That's why the pipeline is the recommended production home for Auto Loader. You declare the ingestion, and the pipeline takes care of the state, ordering and retries.
Here is what that declaration looks like in Python. A function decorated with @dp.table that returns a spark.readStream read of the cloudFiles source defines a streaming table. Notice that there is no schema location and no checkpoint location. In SQL, the equivalent is CREATE OR REFRESH STREAMING TABLE ... AS SELECT * FROM STREAM read_files(...), with the format passed as format => "json".
@dp.table
def customers():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("s3://mybucket/analysis/*/*/*.json")
)Pipelines can also attach expectations to a dataset. An expectation is a SQL boolean constraint, and you choose what happens when a record fails it: warn, drop the record, or fail the update. In the Python API these appear as expect, expect_or_drop and expect_or_fail.
Checkpoint 2 of 5· Check yourself
You are moving a hand-written Auto Loader stream into a Lakeflow pipeline. What should you do with its cloudFiles.schemaLocation and checkpointLocation settings?
Lakeflow pipelines manage the schema location and checkpoint information automatically, so you don't specify either one.
“You do not need to provide a schema or checkpoint location because Lakeflow pipelines automatically manage these settings for your pipelines.”Source: docs.databricks.com
3.Streaming table as the ingestion target
A pipeline can produce three kinds of dataset, and the right target for Auto Loader follows from how each one processes records. Databricks recommends streaming tables for most ingestion use cases, and for files in cloud object storage, Auto Loader gives you incremental, idempotent loading into them.
| Dataset | Processing semantics | Fits when |
|---|---|---|
| Streaming table | Each record processed exactly once, assuming an append-only source | The source grows continuously; you need incremental results, high throughput and low latency |
| Materialized view | Results recomputed as needed to reflect the current state of the data | Aggregations or joins, sources with updates and deletes, or results consumed by many downstream queries |
| View | Evaluated on demand, not persisted | Intermediate steps or expectation checks that don't need publishing to a catalog |
Two refinements matter in practice. First, streaming tables expect append-only input. If records at the source are updated or deleted, use AUTO CDC to apply those changes to the streaming table instead of appending them. Second, a join inside a streaming table doesn't recompute when a dimension changes. That can be fine when speed matters more than exactness. If the result must always be correct, a materialized view recomputes joins when dimensions change. Also, a single pipeline owns each streaming table: no other pipeline can update a table that a pipeline defines, although several flows within that pipeline can append to it.
Streaming ingestion isn't limited to files. Besides Auto Loader for cloud object storage, pipelines can ingest directly from message buses such as Apache Kafka, Azure Event Hubs, Amazon Kinesis and Google Pub/Sub. A Kafka-fed streaming table uses the same @dp.table pattern, with spark.readStream.format("kafka") in place of cloudFiles, or STREAM read_kafka(...) in SQL.
Checkpoint 3 of 5· Exam question
A production Auto Loader stream ingests JSON files into a bronze Delta table. Compliance requires that if the upstream system starts emitting a new column, the pipeline must stop immediately so an engineer can review the schema change before any further rows are written, rather than silently ingesting rows that contain the new field. Which `cloudFiles.schemaEvolutionMode` setting satisfies this requirement?
Correct answer: C — failOnNewColumns
- A. `addNewColumns` updates the schema automatically and lets the stream continue processing with the new column included, which is the opposite of halting for review.
- B. `rescue` keeps the existing schema and routes any new or mismatched fields into a `_rescued_data` column while the stream keeps running, so it does not stop the pipeline.
- C. `failOnNewColumns` stops the stream with an error as soon as a column not present in the known schema is detected, requiring a manual restart after the schema is reviewed, which matches the requirement.
- D. `none` disables schema evolution handling and ignores new columns instead of stopping the stream for them, so it does not enforce a review step.
Checkpoint 4 of 5· Match them up
Match each requirement to the pipeline construct the sources recommend
Tap a term, then the definition that fits it.
Streaming tables suit append-only growth, and AUTO CDC handles a source that changes. Materialized views recompute joins when dimensions change, and views aren't persisted.
“A materialized view keeps its results consistent with the current state of the source data, whereas a streaming table is designed for append-only sources”Source: docs.databricks.com
4.Query changes, bounded state and schema drift
Processing each row once is what makes streaming tables cheap, but it has consequences. A row that is already in the table won't be queried again by later updates. Once you change the logic, the table holds rows produced by different query versions. To reprocess everything you need a full refresh, which re-queries all previous data from the source.
Streaming tables also rely on checkpoint-managed state and expect streams that are bounded. A stream can be naturally bounded, like a directory that stops receiving files, or bounded with a watermark that sets how long to wait for late events. An unbounded stream without a watermark can make the pipeline fail from memory pressure.
Finally, schema drift. Auto Loader supports schema inference and evolution for JSON and CSV in all versions. XML needs Databricks Runtime 14.3 LTS and above, Avro needs 10.4 LTS and above, and Parquet needs 11.3 LTS and above. ORC is unsupported, and Text and Binaryfile have fixed schemas. When new columns appear, the table schema can evolve. Data that doesn't fit, such as a value with a different data type, can be rescued into a JSON blob column for you to inspect later.
That rescue column is named _rescued_data. When you supply your own schema, you can set the rescuedDataColumn option to _rescued_data to capture data that doesn't match it. A pipeline can then police drift with an expectation such as _rescued_data IS NULL, which drops rows that had to be rescued. To have Auto Loader infer real column types for JSON instead of defaulting them, set cloudFiles.inferColumnTypes to true (in SQL, inferColumnTypes => "true" in read_files). If you'd rather fix the schema yourself, pass it with .schema(...), which is required for formats that don't support inference.
Checkpoint 5 of 5· Check yourself
A pipeline ingests ORC files with Auto Loader, and the producer adds a column. What do the sources say about schema evolution for this format?
The schema inference and evolution format table lists ORC as unsupported. "Not applicable (fixed-schema)" is listed for Text and Binaryfile.
“ORC | Unsupported”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Exactly-once ingestion in a Lakeflow pipeline requires you to set cloudFiles.schemaLocation and a checkpointLocation for each Auto Loader source.Why is that wrong?
In Lakeflow pipelines Databricks manages the schema location and checkpoint information automatically. You set them yourself only when you run Auto Loader in plain Structured Streaming.
Covered in Exactly-once through the checkpoint, and what pipelines manage for you
2.Deploying a corrected streaming table query fixes the rows that were already ingested.Why is that wrong?
A streaming table sees each row once. Only rows processed after the change reflect the new logic, and you need a full refresh to reprocess existing rows.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“It provides a Structured Streaming source called cloudFiles.”
↩︎ Auto Loader: the cloudFiles streaming source“Auto Loader can detect schema drifts, notify you when schema changes happen, and rescue data that would have been otherwise ignored or lost.”
↩︎ Auto Loader: the cloudFiles streaming source“This key-value store ensures that data is processed exactly once.”
↩︎ Exactly-once through the checkpoint, and what pipelines manage for you“In case of failures, Auto Loader can resume from where it left off by information stored in the checkpoint location”
↩︎ Exactly-once through the checkpoint, and what pipelines manage for you“The cost of discovering files with Auto Loader scales with the number of files that are being ingested instead of the number of directories”
↩︎ Checkpoint“You do not need to provide a schema or checkpoint location because Lakeflow pipelines automatically manage these settings for your pipelines.”
↩︎ Checkpoint - 2.
“each Auto Loader ingestion workload requires a separate streaming checkpoint.”
↩︎ Exactly-once through the checkpoint, and what pipelines manage for you“Specifying a target directory for the cloudFiles.schemaLocation option enables schema inference and evolution.”
↩︎ Query changes, bounded state and schema drift“If you use Lakeflow pipelines, Databricks manages schema location and other checkpoint information automatically.”
↩︎ Exam trap 1“ORC | Unsupported”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/ldp/conceptsOfficial docs
“While your pipeline runs, it analyzes the dependencies of your defined objects and orchestrates their order of execution and parallelization automatically.”
↩︎ Exactly-once through the checkpoint, and what pipelines manage for you“specify what happens when a record fails: warn, drop the record, or fail the update.”
↩︎ Exactly-once through the checkpoint, and what pipelines manage for you“For files in cloud object storage, Auto Loader provides incremental, idempotent loading.”
↩︎ Streaming table as the ingestion target“pipelines can ingest directly from message buses such as Apache Kafka, Azure Event Hubs, Amazon Kinesis, and Google Pub/Sub.”
↩︎ Streaming table as the ingestion target“Each record is processed exactly one time, assuming an append-only source.”
↩︎ Key concept“A materialized view keeps its results consistent with the current state of the source data, whereas a streaming table is designed for append-only sources”
↩︎ Checkpoint - 4.https://docs.databricks.com/aws/en/ldp/loadOfficial docs
“Databricks recommends using Auto Loader in pipelines for most data ingestion tasks from cloud object storage”
↩︎ Exactly-once through the checkpoint, and what pipelines manage for you - 5.
“To stream source data that changes over time (for example, records that are updated or deleted at the source), use AUTO CDC”
↩︎ Streaming table as the ingestion target“Joins in streaming tables do not recompute when dimensions change.”
↩︎ Streaming table as the ingestion target“An unbounded stream that does not have a watermark can cause a pipeline to fail due to memory pressure.”
↩︎ Query changes, bounded state and schema drift“You can trigger a full refresh to requery all previous data from the source table to update all rows in the streaming table.”
↩︎ Exam trap 2“existing rows will not update to be uppercase, but new rows will be uppercase.”
↩︎ Prediction - 6.
“Use rescuedDataColumn to capture any data that doesn't match the provided schema.”
↩︎ Query changes, bounded state and schema drift - 7.
“Sets inferColumnTypes to true to infer the schema automatically”
↩︎ Query changes, bounded state and schema drift