What you will be able to do
- Configure a file arrival trigger on a Unity Catalog volume or external location and state its requirements
- Configure a table update trigger on one or more tables with Any/All semantics and know its limits
- Tell apart Minimum time between triggers (cooldown) and Wait after last change (debounce)
- Choose between a time-based schedule and a data-driven trigger for a given data-arrival pattern
1.File arrival triggers: run when files land
A schedule runs whether or not new data has arrived. Databricks presents file arrival triggers as the fix for this. They are useful when irregular data arrival makes a scheduled job inefficient. A file arrival trigger starts a run when new files show up in cloud storage, such as Amazon S3, Azure storage or Google Cloud Storage. It checks for new files about once a minute on a best-effort basis, so cloud storage performance can affect the timing. The trigger itself costs nothing beyond the cloud provider's charges for listing files.
No. A file arrival trigger can only watch storage that Unity Catalog governs:
- Location: the root or a subpath of a Unity Catalog external location, or the root or a subpath of a Unity Catalog volume. For example, both /Volumes/mycatalog/myschema/myvolume/ and /Volumes/mycatalog/myschema/myvolume/mydirectory/ are valid.
- Recursive checking: the trigger looks for new files in every subdirectory under the configured path, including nested ones such as subdirC/subdirD.
- Requirements: the workspace must have Unity Catalog enabled. You need READ permission on the storage location and CAN MANAGE permission on the job.
- File events: for best performance, enable file events on the external location. Volumes on such external locations get file event support by default. Enabling file events requires ownership of the external location or the MANAGE privilege on it.
To set it up, click Add trigger, choose File arrival, and enter the URL in Storage location. Optionally, configure the advanced options. Click Test connection to check the setup, then click Save.
Checkpoint 1 of 6· Exam question
An ingestion job is triggered by file arrivals into a landing volume, but the upstream vendor uploads a batch of 200 small files within a 2-3 minute window rather than a single file. Without additional configuration, the job could start multiple times during that window, once for early files and again for later ones. Which setting should the team adjust on the file arrival trigger so the job waits until the batch has fully landed before starting a single run?
Correct answer: B — Increase `wait_after_last_change_seconds` so the run is delayed until a period passes with no new file arrivals, with each new file resetting that countdown.
- A. A post-run cooldown controls how soon the next run can start after a run finishes, but it does not delay the current run to wait for a batch that is still landing, so files arriving mid-batch could still split across runs.
- B. A wait-after-last-change delay resets every time a new file arrives, so it naturally holds off starting a run until the whole rapid batch has finished landing, which is exactly the debouncing behavior the scenario needs.
- C. A fixed cron interval runs on a wall-clock schedule unrelated to when files actually finish arriving, so it could still split a batch across runs or leave a run waiting even when no files are present.
- D. A for-each loop changes how files are processed once a run has already started, but it does nothing to control when the run itself begins relative to a still-arriving batch of files.
Checkpoint 2 of 6· Check yourself
A file arrival trigger watches /Volumes/mycatalog/myschema/myvolume/mydirectory/. A file lands in mydirectory/subdirC/subdirD. What happens?
File arrival triggers look for new files in every subdirectory under the configured location, at any depth.
“A file arrival trigger recursively checks for new files in all subdirectories of the configured location.”Source: docs.databricks.com
Sources1
2.Table update triggers: run when source tables change
A file arrival trigger watches storage. A table update trigger watches tables. It starts a run when a source table changes through an update, merge or delete. You don't need a continuously running cluster, and you don't need to know which process writes to the table.
The trigger can watch these kinds of objects:
- Unity Catalog Delta and Iceberg managed tables - Unity Catalog external tables backed by Delta Lake - Materialized views and streaming tables - Unity Catalog views or metric views that depend on supported tables
To set it up, click Add trigger, choose Table update, and add tables under Tables. If you pick more than one table, use Trigger when to choose whether a run starts when Any table is updated or when All tables are updated. Click Test trigger to check the setup (a file arrival trigger uses Test connection instead), then click Save. You can also define the trigger through the Jobs API with jobs/create, jobs/update or jobs/reset.
Limits to know:
- A trigger can watch up to 10 managed or Delta tables.
- For a view, its source tables count toward that limit of 10. A view that depends on 11 tables can't be used.
- A watched view can depend on at most 10 other views.
- A view counts as updated when any of its source tables changes. As a result, a run can start for changes that the view's definition filters out.
- Views that use read_files, depend on tables outside Unity Catalog, or depend on federated tables are not supported.
A table update trigger also gives the job's parameters some dynamic value references. For example, {{job.trigger.table_update.updated_tables}} returns a JSON list of the tables updated since the last run. Other references return each table's latest commit timestamp and version. Running only on fresh data and knowing which tables changed lets a task skip work on tables that haven't moved.
Checkpoint 3 of 6· Check yourself
A job should start only after both the orders table and the customers table have been refreshed. How do you configure the table update trigger?
One trigger can watch several tables. Setting Trigger when to All tables are updated holds the run until every watched table has changed.
“When selecting multiple tables, you can specify if a job is triggered when any or all of the tables are updated.”Source: docs.databricks.com
Sources2
3.Cooldown vs debounce: the two advanced options
File arrival and table update triggers have the same two advanced options. Both control how incoming data turns into job runs.
| Option | Pattern | Timer starts | Use it to |
|---|---|---|---|
| Minimum time between triggers in seconds | Cooldown | After the previous run completes | Cap how often runs are created, at most one per interval |
| Wait after last change in seconds | Debouncing | At the most recent arrival; each new arrival resets it | Process a whole batch in one run once it has finished landing |
For file arrival, the docs give these examples:
- Cooldown only. Minimum time 900 means at most one run every 15 minutes, even if files keep arriving. - Debounce only. If a batch starts about every 5 minutes, Wait after last change 60 means a run starts after 60 seconds with no new files. The wait must be longer than the gaps between files within a batch, so the timer doesn't run out mid-batch. It must also be shorter than the gap between batches, so two batches don't merge into one run. - Both together. The trigger waits for each batch to finish landing and still starts at most one run per 15 minutes.
The table update docs work through the combined case. Minimum time is 120 seconds and Wait after last change is 60 seconds. No run starts before 120 seconds have passed. If updates arrive at 5 and 115 seconds, the run doesn't start until after 175 seconds, which is 60 seconds after the last update.
Checkpoint 4 of 6· Exam question
A data engineering team ingests raw CSV files from an external vendor at unpredictable times, sometimes several times per hour and sometimes not for a full day. They want the Lakeflow job to start processing as soon as new files land in a Unity Catalog volume, without running on a fixed cron schedule that would waste cluster time when no files have arrived. Which trigger configuration best fits this requirement?
Correct answer: B — Configure a file arrival trigger on the Unity Catalog volume so the job automatically starts a run once Databricks detects that new files have been written there.
- A. An hourly cron schedule would run the job on a fixed cadence even during hours with no new files, wasting cluster time, and could also leave files unprocessed for up to an hour after they arrive. It does not react to the actual event the team cares about.
- B. A file arrival trigger watches the monitored Unity Catalog volume and starts a run once new files are detected there, which directly matches an unpredictable, event-driven arrival pattern without needing a fixed schedule.
- C. Watching the job's own output table reacts to the job having already run, not to new vendor files landing in the source volume, so it would not help the job start in response to new source data.
- D. A continuous trigger reruns the job repeatedly regardless of whether new data exists, which burns compute continuously instead of only reacting when the vendor actually delivers files.
Checkpoint 5 of 6· Check yourself
Which statement correctly describes Minimum time between triggers in seconds?
Minimum time between triggers is a cooldown that starts after a run completes. The resetting timer described in the first option is Wait after last change.
“Limits the job to at most one run per this interval (a cooldown between runs).”Source: docs.databricks.com
4.Choosing a trigger and processing what arrived
Choose the trigger that matches how the data arrives:
- Data arrives at a known time (for example, a nightly extract) or the business wants output at fixed times: use a scheduled trigger. - Files land irregularly in Unity Catalog storage: use a file arrival trigger. Databricks recommends it over a continuous pipeline for event-driven processing, because the cluster runs only when there are new files to process. - The job depends on tables that other pipelines update: use a table update trigger. It runs when new data is ready, and you don't need to know which process updates the table. - An always-on streaming workload: use continuous mode.
The trigger starts the run, but the job still has to work out which files are new. For that, the docs recommend Auto Loader, which processes new files incrementally and exactly once. In the sample below, the notebook reads the same path that the file arrival trigger watches. It uses availableNow so each triggered run processes whatever has arrived and then stops.
Checkpoint 6 of 6· Fill the gap
This notebook runs from a file arrival trigger. Which trigger setting makes each run process the files available and then stop?
streamingQuery = spark.readStream.format("cloudFiles") \
.option("cloudFiles.format", "json") \
.option("cloudFiles.schemaLocation", checkpoint_location) \
.option("cloudFiles.useManagedFileEvents","true") \
.load(file_location) \
.writeStream \
.option("checkpointLocation", checkpoint_location) \
.trigger( ? = True) \
.toTable(sink_table)The documented sample uses availableNow = True. Each job run handles the newly arrived files and then finishes, and the checkpoint gives exactly-once processing.
Source: docs.databricks.comExam traps
Each one states something that sounds right. Open it to see what is actually true.
1.A file arrival trigger can watch any cloud storage path, or only the exact directory you give it.Why is that wrong?
It must watch the root or a subpath of a Unity Catalog external location or volume, and it checks all subdirectories recursively.
Covered in File arrival triggers: run when files land
2.Minimum time between triggers and Wait after last change are the same thing under two names.Why is that wrong?
Minimum time is a cooldown counted from the end of the previous run. Wait after last change is a debounce that resets on every new arrival.
3.A table update trigger on a view fires only when rows that the view returns change.Why is that wrong?
The trigger treats a view as updated when any of its source tables changes. A run can therefore start for changes that the view filters out.
Covered in Table update triggers: run when source tables change
Practise it for real
Make a job run automatically when JSON files land in a Unity Catalog volume, and load each batch into a Delta table with Auto Loader.
1.Create a notebook from the documented Auto Loader snippet. Set file_location to your volume path, checkpoint_location to a separate path outside it, and sink_table to a Delta table.
Why: The trigger only starts the run. Auto Loader and its checkpoint work out which files are new and process them exactly once.
You should see: A notebook whose stream uses .trigger(availableNow = True) and writes with .toTable(sink_table).
2.Create a job with that notebook as a task. In the Job details pane, click Add trigger and choose File arrival.
Why: File arrival is the data-driven trigger for files landing in Unity Catalog storage.
You should see: The trigger dialog shows a Storage location field.
3.Enter the same volume path in Storage location. Under the advanced options, set Wait after last change in seconds to 60.
Why: The debounce holds the run until a burst of files has finished landing.
You should see: Both fields are filled in.
4.Click Test connection, then click Save.
Why: Test connection checks that you can read the location before the trigger goes live.
You should see: The test succeeds, and the trigger appears under Schedules & Triggers.
5.Copy a few JSON files into the volume, then watch the job's runs.
Why: The trigger checks for new files about once a minute on a best-effort basis.
You should see: A run starts a minute or two after the last file lands, and the new rows appear in the sink table.
Stuck? Get a nudge
If no run starts, check that you have READ on the volume and CAN MANAGE on the job, and that the trigger is Active, not Paused.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“This feature is useful when a scheduled job's efficiency is compromised by irregular new data arrivals.”
↩︎ File arrival triggers: run when files land“File arrival triggers make a best effort to check for new files every minute”
↩︎ File arrival triggers: run when files land“You must have READ permission on the storage location and CAN MANAGE permissions on the job.”
↩︎ File arrival triggers: run when files land“To validate the configuration, click Test connection.”
↩︎ File arrival triggers: run when files land“the trigger waits until a batch finishes landing (60 seconds with no new files) before starting a run”
↩︎ Cooldown vs debounce: the two advanced options“Auto Loader incrementally and efficiently processes new files with exactly-once guarantees.”
↩︎ Choosing a trigger and processing what arrived“monitor the root of a Unity Catalog external location or volume, or a subpath of an external location or volume”
↩︎ Exam trap 1“each new arrival resets the timer (debouncing).”
↩︎ Exam trap 2“A file arrival trigger recursively checks for new files in all subdirectories of the configured location.”
↩︎ Checkpoint“Use this option when files arrive in batches and you want to process the whole batch in a single run”
↩︎ Prediction“Limits the job to at most one run per this interval (a cooldown between runs).”
↩︎ Checkpoint - 2.
“Use this feature to run a job when new data is ready without the need for a continuously running cluster”
↩︎ Table update triggers: run when source tables change“To validate the configuration, click Test trigger.”
↩︎ Table update triggers: run when source tables change“You can select up to 10 managed or Delta tables per trigger.”
↩︎ Table update triggers: run when source tables change“A JSON list of tables updated since the last job run.”
↩︎ Table update triggers: run when source tables change“then it does not trigger the run until at least 120 seconds has elapsed”
↩︎ Cooldown vs debounce: the two advanced options“It is possible for a job run to be triggered for data changes that are filtered out by the view definition.”
↩︎ Exam trap 3“When selecting multiple tables, you can specify if a job is triggered when any or all of the tables are updated.”
↩︎ Checkpoint - 3.https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/file-events-explainedOfficial docs
“your cluster only runs when there are new files to process.”
↩︎ Choosing a trigger and processing what arrived