Lakeflow Pipelines (formerly DLT) for Streaming Inference: Streaming Tables and Pipeline Modes

1.Delta Live Tables is now Lakeflow pipelines

The exam objective still says "Delta Live Tables", but the documentation uses a newer name. Databricks has renamed the product, and existing code does not need to be migrated. DLT code keeps working as it is. The new names matter because current examples, including the ones that apply MLflow models, use the new Python module and decorators.

Switching to dp changes the decorators too. @dlt becomes @dp, @view becomes @temporary_view, and table creation is now split into two decorators. @table creates streaming tables and the new @materialized_view creates materialized views. Some traces of the old name remain: the classic SKUs still start with DLT, event log schemas with dlt in the name have not changed, and the old Python APIs still run, although Databricks recommends the new names.

Sources1

2.Where streaming predictions land: streaming tables vs. materialized views

A pipeline writes to three kinds of dataset, and each one processes records differently. That choice decides whether your inference step scores only new records or recomputes everything. A streaming table processes each record once from an append-only source. A materialized view is recomputed as needed so that it reflects the current state of the data. A view is evaluated on demand and not persisted. All tables a pipeline creates and manages are Delta tables, with Delta Lake guarantees such as ACID transactions and time travel.

For streaming inference, the streaming table is the natural fit. The docs recommend one when a query reads from a data source that keeps growing, when results should be computed incrementally, and when the pipeline needs high throughput and low latency. On each update, the flows that feed a streaming table read only the changed information in the streaming source and append the new results. A streaming table must read from a streaming source, such as files loaded with Auto Loader or a message bus like Apache Kafka, Azure Event Hubs or Google Pub/Sub.

@dp.table
def streaming_bronze():
  return (
    # Since this is a streaming source, this table is incremental.
    spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "json")
      .load("s3://path/to/raw/data")
  )

@dp.table
def streaming_silver():
  # Since we read the bronze table as a stream, this silver table is also
  # updated incrementally.
  return spark.readStream.table("streaming_bronze").where(...)

@dp.materialized_view
def live_gold():
  # This table will be recomputed completely by reading the whole silver table
  # when it is updated.
  return spark.read.table("streaming_silver").groupBy("user_id").count()

Streaming tables have one condition attached. They are built for streams over bounded state, meaning streams that end naturally or that are bounded by a watermark. An unbounded stream with no watermark can make the pipeline fail from memory pressure. If the source data is updated or deleted rather than only growing, or the query performs joins or aggregations that must stay correct, the docs point you to a materialized view instead.

Sources23

3.How often predictions refresh: triggered vs. continuous mode

Running a streaming table does not by itself make the pipeline run all the time. A pipeline runs in either triggered or continuous mode. The mode controls when new predictions appear. In triggered mode, the pipeline processes the data that is available when the update starts and then stops. In continuous mode, it keeps processing new data as it arrives. To avoid wasted work, a continuous pipeline monitors the Delta tables it depends on and runs an update only when their contents have changed.

How you get continuous execution matters too. Databricks recommends wrapping the pipeline in a continuous job instead of setting the pipeline's own Pipeline mode to continuous. The job controls the pipeline's lifecycle and unlocks serverless performance modes, such as Standard mode, that the built-in continuous setting does not support. A job's execution mode also overrides the pipeline's setting, so the docs advise leaving Pipeline mode at the default, triggered, when a continuous job wraps the pipeline. In continuous runs, pipelines.trigger.interval controls how often each flow starts an update. Standalone streaming tables, created outside a Lakeflow pipeline, always refresh in triggered mode. For sub-second, end-to-end latency, the docs also describe a separate real-time mode.

Sources4

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.In the dp API, @dp.table creates a generic table that may be batch or streaming, the same way @dlt.table did.Why is that wrong?

   In Lakeflow pipelines, @table creates streaming tables. Materialized views now have their own decorator, @materialized_view.

   Covered in Delta Live Tables is now Lakeflow pipelines

2. 2.Streaming inference with a streaming table requires the pipeline to run in continuous mode.Why is that wrong?

   Pipeline mode and dataset type are independent. A streaming table can be refreshed by a triggered pipeline, which processes only the data available when the update starts.

   Covered in How often predictions refresh: triggered vs. continuous mode

3. 3.The recommended way to run a pipeline continuously is to set its Pipeline mode to continuous.Why is that wrong?

   Databricks recommends a continuous job instead. The job controls the execution mode and overrides the pipeline's own setting.

   Covered in How often predictions refresh: triggered vs. continuous mode