Table Update Triggers and Rate Control for Pipeline Dependencies

1.Expressing upstream table dependencies with a table update trigger

A pipeline dependency between jobs usually means that job B should process what job A wrote, as soon as A has written it. A Scheduled trigger can only approximate this by leaving a safety margin after A usually finishes. A table update trigger starts a Lakeflow job when monitored Unity Catalog tables change, so it expresses the dependency directly: point it at the tables that A produces.

When B reads several upstream tables, the Trigger when option defines when the data counts as ready. Choose All tables are updated if B's output only makes sense once every input is refreshed, such as a join across two feeds. Choose Any table is updated if each input is useful on its own. One trigger can monitor up to 10 managed or Delta tables.

The trigger also passes along what changed. Three dynamic value references become available as job parameter values, so downstream tasks can process just the new commits instead of rescanning everything.

Views need some care: - A trigger on a Unity Catalog view actually watches the view's source tables, and those tables count toward the 10-table limit. - Any change to a source table counts as an update to the view, so a run can fire for changes that the view's definition filters out. - Views that use read_files, depend on tables outside Unity Catalog, or depend on federated tables aren't supported. Creating the trigger still succeeds, but updates to those dependencies never start a run.

Sources1

2.Cooldown and debouncing: controlling how often data triggers fire

Data-driven triggers bring a problem that schedules don't have: data can arrive too often, or in pieces. File arrival and table update triggers both have two advanced options that control how changes turn into runs.

The file arrival documentation gives worked examples: - Minimum time between triggers of 900 limits the job to one run per 15 minutes. - If a new batch of files starts about every 5 minutes, Wait after last change of 60 starts a run only after 60 seconds with no new files. This assumes files within a batch arrive less than 60 seconds apart, and batches are more than 60 seconds apart. - If you set both, the trigger waits for each batch to finish landing and never starts more than one run per 15 minutes.

The table update documentation describes using both options as a sequence: first the minimum time, then the wait after the last change. With a 120-second minimum and a 60-second wait, updates at 5 seconds and 115 seconds mean no run starts until after 175 seconds.

Sources12

3.Processing exactly what the file trigger found

A file arrival trigger starts the job, but it doesn't give the job a list of the new files. The documented pattern is to pair the trigger with Auto Loader in a notebook task. Point Auto Loader at the same URL as the trigger, and put its checkpoint location outside that path. Auto Loader finds what's new since the last run and processes it exactly once. A bounded streaming trigger makes the query stop once the backlog is processed, so each job run finishes cleanly and the next one starts on the next file arrival.

streamingQuery = spark.readStream.format("cloudFiles") \
  .option("cloudFiles.format", "json") \
  .option("cloudFiles.schemaLocation", checkpoint_location) \
  .option("cloudFiles.useManagedFileEvents","true") \
  .load(file_location) \
  .writeStream \
  .option("checkpointLocation", checkpoint_location) \
  .trigger( ?  = True) \
  .toTable(sink_table)

Sources2

4.When to run continuously instead, and the overall decision

If data arrives constantly and latency matters more than cost, Databricks recommends a Continuous trigger for always-on streaming workloads. The trade-off is the same one described for pipelines. Continuous processing needs an always-running cluster, which costs more but reduces latency. Triggered processing runs only as long as each update takes.

Continuous jobs also have hard limits: - Only one instance of a continuous job can run at a time. - The gap between one run ending and the next starting should be under 60 seconds. - You can't set retry policies. Instead, the whole job retries automatically on failure, with exponential backoff. - You can't use task dependencies.

So a multi-task DAG whose tasks must run in order needs a triggered job, either scheduled or data-driven.

Sources34

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Minimum time between triggers waits until a batch of files has finished landing.Why is that wrong?

   Waiting for a batch to finish is what Wait after last change does: it resets its timer on every new arrival. Minimum time between triggers is a cooldown that limits how often runs can start after a run completes.

   Covered in Cooldown and debouncing: controlling how often data triggers fire

2. 2.A continuous job is the lowest-latency way to run a multi-task pipeline with ordered dependencies.Why is that wrong?

   Continuous jobs don't support task dependencies. A multi-task DAG with ordered tasks needs a triggered job, such as one with a file arrival or table update trigger.

   Covered in When to run continuously instead, and the overall decision