What you will be able to do
- Choose a Structured Streaming trigger that turns the same query into scheduled incremental batch or always-on processing
- Weigh the cost and latency of availableNow, processingTime and real-time mode for a given freshness requirement
- Tell apart the Jobs continuous trigger, pipeline continuous mode and the Structured Streaming Continuous trigger
1.The trigger decides how often a stream checks for new data
A Structured Streaming query always processes data incrementally. Its trigger decides how often it checks for new data. The same query, with the same checkpoint, can run as a near-real-time stream, as a micro-batch every few minutes, or as a once-a-day incremental batch. Changing one line, .trigger(...), moves a workload between those shapes. Auto Loader runs on Structured Streaming too, so the same choice decides how often files are ingested and how much that costs.
| Trigger mode | Python syntax | Best for |
|---|---|---|
| Unspecified (default) | N/A | General-purpose streaming with 3-5 second latency; equivalent to processingTime with 0 ms intervals |
| Processing Time | .trigger(processingTime='10 seconds') | Balancing cost and performance by not checking for data too often |
| Available Now | .trigger(availableNow=True) | Scheduled incremental batch processing of everything available when the job is triggered |
| Real-time mode | .trigger(realTime='5 minutes') | Sub-second operational workloads |
Leaving the trigger unset is still a choice, and often an expensive one. With no trigger, the stream starts a new micro-batch as soon as the last one finishes, so it never stops checking the source. Databricks recommends choosing a trigger that balances latency against cost. Without one, the frequent listing and reading can show up as unexpected storage costs from your cloud provider.
Checkpoint 1 of 3· Check yourself
A team runs an Auto Loader stream with no .trigger(...) set. They think this is the cheapest setup because the stream only does work when new files arrive. What does the stream actually do?
With no trigger set, the stream behaves like processingTime with a 0 ms interval. It never stops checking the source. This gives the lowest micro-batch latency, but the frequent listing and reading raises cloud storage API costs, so the setup is not cheap.
“This delivers the lowest micro-batch latency, but it increases cloud storage API costs.”Source: docs.databricks.com
Sources1
2.Incremental batch versus always-on: what each trigger costs
availableNow turns a stream into an incremental batch job. When triggered, it processes everything that has arrived since the last checkpoint and then stops. Compute runs only for that window. Schedule it hourly or nightly from a Lakeflow Job and you get batch-like cost with streaming bookkeeping: no data is reprocessed and none is missed. On serverless notebooks and jobs, Databricks recommends this trigger, and it is one of only two that are allowed there.
processingTime keeps the stream running and sets the pace. An interval such as '10 seconds' cuts overhead because the query checks the source less often. An interval of 0 starts each micro-batch as soon as the previous one finishes, provided new data is available. That gives the lowest micro-batch latency, but it raises cloud storage API costs. Use it for operational workloads, which should not use AvailableNow, Once or the Continuous trigger.
Real-time mode is for latency that micro-batches cannot reach. Micro-batch mode suits ETL and analytics that need results within seconds or minutes. Fraud detection and personalisation need sub-second results, and real-time mode can deliver end-to-end latency as low as five milliseconds. In practice the trigger choice follows the freshness requirement. A daily or hourly report needs availableNow on a schedule. Minute-level ETL needs a processingTime interval. Sub-second action needs real-time mode, and compute has to stay running the whole time.
Checkpoint 2 of 3· Match them up
Match each trigger setting to the workload it fits at the lowest cost that still meets the freshness requirement.
Tap a term, then the definition that fits it.
availableNow processes the backlog and then stops, so a scheduled run costs about as much as a batch job. A 1-minute interval sets the pace and cuts how often the source is checked. A 0 interval gives the lowest micro-batch latency but raises storage API costs. Sub-second needs go beyond micro-batches, so they need real-time mode.
“This delivers the lowest micro-batch latency, but it increases cloud storage API costs.”Source: docs.databricks.com
3.Three different settings called continuous
"Continuous" names three separate settings on Databricks, and exam scenarios rely on the confusion. The first is the Lakeflow Jobs continuous trigger, a scheduling feature rather than a Structured Streaming trigger interval. Databricks recommends it for streaming jobs. It prevents a second concurrent run, starts a new run when one fails and backs off exponentially between retries. The second is continuous pipeline mode. The third is the Structured Streaming Continuous trigger, which should not be used for operational workloads.
Pipelines repeat the trigger trade-off at the pipeline level. A triggered pipeline updates every table from the data available when the update starts, then stops, so compute runs only during the update. It fits refreshes every 10 minutes, hourly or daily. A continuous pipeline handles data as it arrives and suits freshness of 10 seconds to a few minutes. It needs a cluster that is always running, and that is usually the largest cost factor. Start with triggered, and move to continuous only once a latency requirement is proven.
For a pipeline that must run always-on, Databricks recommends a continuous job rather than setting Pipeline mode to continuous. When a job orchestrates a pipeline, the job's execution mode overrides the pipeline's setting. A continuous job runs the pipeline continuously even when Pipeline mode is triggered, and a scheduled job runs a single update even when it is set to continuous. The continuous job also unlocks serverless performance modes such as Standard mode. Leave Pipeline mode at its default, triggered, so the pipeline doesn't behave unexpectedly when someone runs it outside the job.
Checkpoint 3 of 3· Check yourself
Your Lakeflow pipeline's Pipeline mode is left at its default, triggered. You then wrap the pipeline in a Lakeflow Job that uses the continuous trigger. How does the pipeline run?
When a job orchestrates a pipeline, the job's execution mode wins, so a continuous job runs the pipeline continuously even if Pipeline mode is triggered. The Jobs continuous trigger is a scheduling feature. It does not change any Structured Streaming trigger, and Databricks recommends leaving Pipeline mode at triggered in this setup.
“A continuous job runs its pipeline continuously even if the pipeline's Pipeline mode is triggered”Source: docs.databricks.com
Sources
(4)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Trigger intervals control how frequently Structured Streaming checks for new data.”
↩︎ The trigger decides how often a stream checks for new data“Equivalent to processingTime trigger with 0 ms intervals.”
↩︎ The trigger decides how often a stream checks for new data“Otherwise, you might see unexpected storage costs from your cloud provider.”
↩︎ The trigger decides how often a stream checks for new data“Scheduled incremental batch processing. Processes as much data as is available at the time the streaming job is triggered.”
↩︎ Incremental batch versus always-on: what each trigger costs“Balancing cost and performance. Reduces overhead by preventing the system from checking for data too frequently.”
↩︎ Incremental batch versus always-on: what each trigger costs - 2.
“This delivers the lowest micro-batch latency, but it increases cloud storage API costs.”
↩︎ Incremental batch versus always-on: what each trigger costs“Don't use AvailableNow, Once, or Continuous for operational workloads.”
↩︎ Incremental batch versus always-on: what each trigger costs“RTM is up to 100x faster than micro-batch mode, with end-to-end latency as low as five milliseconds.”
↩︎ Incremental batch versus always-on: what each trigger costs“Starts a new run when a previous run fails.”
↩︎ Three different settings called continuous“This refers to the Databricks Jobs scheduling feature, not the Structured Streaming trigger interval.”
↩︎ Exam trap 1“This delivers the lowest micro-batch latency, but it increases cloud storage API costs.”
↩︎ Exam trap 3“This delivers the lowest micro-batch latency, but it increases cloud storage API costs.”
↩︎ Checkpoint“For notebooks and jobs on serverless compute, only Trigger.AvailableNow() and Trigger.Once() are supported.”
↩︎ Prediction - 3.
“Continuous pipelines require an always-running cluster, which is more expensive but reduces processing latency.”
↩︎ Three different settings called continuous“A continuous job runs its pipeline continuously even if the pipeline's Pipeline mode is triggered”
↩︎ Three different settings called continuous“Databricks recommends running continuous pipelines with a continuous job rather than setting the value of Pipeline mode to continuous.”
↩︎ Exam trap 2 - 4.
“Continuous mode keeps compute running to process new data with minimal delay, which is usually the largest cost factor”
↩︎ Three different settings called continuous