What you will be able to do
- Find the node that broke a dbt invocation using the run summary, run_results.json and logs/dbt.log
- Predict which downstream models are skipped, run anyway, or cancelled when a model fails, under on_error and --fail-fast
- Rerun only the broken part of the DAG with graph operators and the result: selector
- Trace a failed dbt Cloud job or a warehouse-side Database Error back to its cause
Key concept
Failure point and skipped descendants — When a model fails, it is the failure point for that part of the DAG. By default dbt skips every model downstream of it, so one root cause can show up as many unbuilt models. Find and fix the failing node first, then rebuild it together with its descendants.
1.Locating the node that broke the run
When a dbt invocation fails partway through, the project ends up in a mixed state. Some models were built, at least one errored, and others may never have started. Troubleshooting starts with finding the one node that actually broke. Everything downstream of it is a symptom, and fixing symptoms only wastes reruns.
The error message is the first clue. dbt's error output will normally contain the type of error and the file where it happened. Open that file and check whether the fix is obvious. If it isn't, isolate the problem by running one model at a time, or by undoing the change that broke things. Reading the error types themselves (Runtime, Compilation, Dependency, Database) is covered in the lesson on [logged error messages](lesson:3.1).
Completed with 1 error and 0 warnings:
Runtime Error in model customers (models/customers.sql)
Parser Error: syntax error at or near "selct"
Done. PASS=4 WARN=0 ERROR=1 SKIP=0 TOTAL=5Read the Done line before anything else. ERROR=1 means one node failed. SKIP counts the nodes dbt never attempted because something upstream of them failed. In this run, orders still finished OK even though customers errored, because the two models ran independently. A failure only stops the models that are downstream of it.
Three artifacts hold the details. run_results.json has one entry per node, with a status and a unique_id that links the result back to the node in the manifest. However, only executed nodes appear there, so a model that was never attempted has no entry. logs/dbt.log records every query dbt ran, with the most recent errors at the bottom. target/run holds the exact SQL dbt executed for each model, which is useful when the failure comes from the warehouse (see [compiled code](lesson:3.2)).
Checkpoint 1 of 8· Check yourself
After a failed dbt run, you search run_results.json for a model that sits far downstream of the failure. What will you find?
run_results.json lists only the nodes that actually executed. A model that was never attempted because something upstream failed does not appear, so look for its failed ancestor instead.
“Note that only executed nodes appear in the run results.”Source: docs.getdbt.com
2.How a failure propagates downstream
What happens downstream of a failed model is controlled by the model's on_error config. The default is skip_children: every model downstream of the failed one is skipped. This is why one broken staging model can leave a whole mart layer unbuilt. The fix belongs at the failing node, not at each skipped model. The other value is continue, which lets downstream models run anyway. Use it when they can still produce meaningful output, for example because they have fallback logic or independent data sources. on_error applies only to models. Whether downstream models run after a test failure depends on the test's severity config.
--fail-fast (-x) works across the whole invocation. With this flag, dbt exits at the first failure, whether that failure is a model error or a test failure. If other models are still running when it happens, dbt terminates their connections and cancels them. --fail-fast also overrides on_error: continue. In a fail-fast run, the first error stops everything that has not finished yet.
| Setting | Downstream models | Overall run |
|---|---|---|
| on_error: skip_children (default) | Skipped | Fails |
| on_error: continue | Run anyway | Still fails; the failed model stays an error |
| --fail-fast with on_error: continue | Remaining models skipped; dbt stops at the first failure | Fails |
Checkpoint 2 of 8· Match them up
Match each setting to its effect when a model fails
Tap a term, then the definition that fits it.
skip_children is the default. continue only changes whether the children run, not whether the run fails. --fail-fast overrides both and stops the whole invocation.
“The --fail-fast flag takes precedence over on_error: continue.”Source: docs.getdbt.com
Checkpoint 3 of 8· Exam question
A nightly `dbt build` run fails partway through: `stg_orders` returns a Database Error and dbt marks `int_orders`, `fct_orders`, and their tests as SKIPPED. After fixing the SQL bug in `stg_orders`, which command re-executes only the previously failed model and everything that depends on it, without re-running models that already succeeded?
Correct answer: A — Run `dbt build --select result:error+` against the same target so only the model that errored and its downstream dependents rebuild from this run's results.
- A. The `result:error` method reads the prior invocation's run results and selects nodes that errored, and the trailing `+` extends selection to everything downstream, so exactly the failed model and its skipped dependents rebuild.
- B. A full refresh rebuilds every incremental model project-wide regardless of whether it failed, which wastes warehouse compute on models that already succeeded and were never affected by the error.
- C. Rerunning only the fixed model leaves `int_orders`, `fct_orders`, and their tests in a skipped state, so the DAG is still incomplete after this command finishes.
- D. `state:modified` compares SQL and config against a prior manifest to find code changes; it says nothing about which nodes errored or were skipped in the failed run, so it would miss unrelated failures.
3.Rerunning only the broken part of the DAG
After you fix the failing model, you usually don't need to rebuild the whole project. What still needs to run is the failed model and everything that was skipped because of it. In graph terms, that is the model and all its descendants. Placing the + operator after a model name selects exactly that set.
dbt run --select "my_model+" # select my_model and all descendants
dbt run --select "+my_model" # select my_model and all ancestors
dbt run --select "+my_model+" # select my_model, and all of its ancestors and descendantsTyping model names by hand stops working once several models fail in one run. The result:<status> method selects nodes by their outcome in the previous run's artifacts. result:error+ selects every errored model and its descendants. Combine it with state:modified+ (explained in [state selection](lesson:7.1)) and one command rebuilds both the broken models and anything you changed while fixing them. --defer lets unselected upstream references resolve to production.
Checkpoint 4 of 8· Fill the gap
Complete the command that rebuilds and retests every errored model and its downstream models, along with your modified models.
dbt build --select state:modified+ ? --defer --state path/to/prod/artifactsresult:error selects the models that errored in the previous run. The trailing + adds their descendants, which were skipped the first time.
dbt retry automates the same idea. It reads run_results.json and continues from the point of failure, without running the upstream models that already succeeded (details in [dbt retry](lesson:7.2)). It has one blind spot. If a run fails before any node executes, for example because the warehouse connection or permissions fail, retry has no recorded nodes to resume from. In that case, fix the connection and rerun the full job.
Checkpoint 5 of 8· Exam question
A scheduled `dbt build` job fails when `fct_revenue` throws a Database Error, and the run stops before three downstream models execute. Which statement about running `dbt retry` immediately afterward is accurate?
Correct answer: A — `dbt retry` reads the `run_results.json` artifact from the failed invocation and resumes execution starting at the nodes that did not complete successfully.
- A. This describes the documented mechanism: dbt retry inspects the run results from the previous invocation and picks up execution at the nodes that errored or never ran, skipping ones that already succeeded.
- B. Rerunning every node defeats the purpose of a retry command; nodes that already completed successfully in the prior invocation are left alone rather than being recompiled and re-executed.
- C. The retry command does not require repeating the original selection flags on the command line; it derives the set of nodes to rerun from the stored run results rather than from a re-typed `--select` argument.
- D. Deleting the target directory would remove the very run-results artifact that the retry command depends on, so this description contradicts how the command actually determines what to rerun.
4.Failures surfacing through jobs and the warehouse
In production, failures reach you through the scheduler, not your terminal. Set up dbt notifications for failed runs, then debug those runs the same way you would debug runs in development: read the error, find the node, and isolate it. In dbt, the errored run's page shows each command as a separate step, and the step that failed is marked with ❌.
Checkpoint 6 of 8· Put it in order
A scheduled dbt job ended in Error. Put the recovery steps in order.
- 1.Choose the job run that errored
- 2.Click Rerun, then choose Rerun from failure
- 3.Fix the cause, then save and commit it to Git
- 4.Expand the failed step in the Run Summary tab
- 5.Open Deploy, then Run History
Find the failed step first, then commit the fix so the rerun picks it up. Rerun from failure runs only the failed step and the steps that were skipped after it.
“Select Deploy from the top navigation bar and choose Run History.”Source: docs.getdbt.com
Choose between the two rerun options based on what the fix changed. Rerun from failure opens a modal that lists only the failed step and any skipped steps, and the resulting run carries the banner "This run resumed execution from last failed step". Rerun from start runs every step again. Use it when the fix affects steps that already passed. Both options need dbt 1.6 or newer and a project that dbt can still parse.
Database Errors are where integrated tools make diagnosis hardest. The warehouse produces the message and dbt passes it on unchanged, so the warehouse's own documentation may be the place that explains it. Most of the time the cause is in the model's SQL. Some errors, though, come from queries dbt runs behind the scenes: introspective queries, schema creation, pre-hooks and post-hooks, on-run-start and on-run-end hooks, and the merge, update and insert statements used by incremental models and snapshots. None of these appear in the model file. To find them, read logs/dbt.log, which records every query dbt ran. To get a clean log, empty the file and rerun only the failing model.
Checkpoint 7 of 8· Check yourself
An incremental model fails with a Database Error, and its compiled select statement runs fine in the warehouse console. Where should you look next?
Merge, update and insert statements for incremental models run behind the scenes and are not part of the compiled select. They show up in the log, which records every query dbt ran.
“this contains all the queries dbt has run.”Source: docs.getdbt.com
Checkpoint 8 of 8· Exam question
A teammate pastes this end of a `dbt build` log and asks why three models show `SKIP` instead of running: ``` 16:42:03 1 of 7 START sql table model staging.stg_customers ........... [RUN] 16:42:04 1 of 7 ERROR creating sql table model staging.stg_customers ... [ERROR in 0.61s] 16:42:04 2 of 7 SKIP relation intermediate.int_customer_orders ......... [SKIP] 16:42:04 3 of 7 SKIP relation marts.fct_customers .................... [SKIP] 16:42:04 4 of 7 SKIP relation marts.dim_customers .................... [SKIP] ``` What is the most accurate explanation for the three `SKIP` lines?
Correct answer: A — `int_customer_orders`, `fct_customers`, and `dim_customers` all depend on `stg_customers` through `ref()`, so dbt automatically skips any node downstream of a node that errored.
- A. dbt builds nodes in dependency order and, when a node errors, marks every node reachable downstream of it as skipped rather than attempting to build on top of a table that was never created.
- B. `--fail-fast` is an opt-in flag, not a default, and even without it dbt still skips only the descendants of the failed node rather than every remaining node in the entire invocation.
- C. The log shows a Database Error during table creation, not a Jinja compilation failure, and shared macro usage alone does not create a `ref()` dependency edge that would cause automatic skipping.
- D. This log is from a local `dbt build` invocation with no scheduler involved; the skip behavior comes from dbt's own dependency-aware execution engine, not an external job orchestrator cancelling steps.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Setting on_error: continue on a model makes the run succeed as long as its downstream models build.Why is that wrong?
continue only stops dbt from skipping the children. The failed model is still recorded as an error, and the run still fails.
Covered in How a failure propagates downstream
2.--fail-fast stops only on model errors, so a failing test still lets the rest of the build continue.Why is that wrong?
fail_fast is a global flag. It stops at the first run error or test failure and cancels models that are still running.
Covered in How a failure propagates downstream
3.dbt retry can always recover a failed run, even one that died on a warehouse connection error before any model ran.Why is that wrong?
Retry resumes from the nodes recorded in run_results.json. If nothing executed, it has nothing to resume, and you have to rerun the full job.
Covered in Rerunning only the broken part of the DAG
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.getdbt.com/guides/debug-errorsOfficial docs
“Recent errors will be at the bottom of the file.”
↩︎ Locating the node that broke the run“will normally contain the type of error”
↩︎ Locating the node that broke the run“by running one model a time”
↩︎ Locating the node that broke the run“These errors come from your data warehouse, and dbt passes the message on.”
↩︎ Failures surfacing through jobs and the warehouse“For incremental models, and snapshots: merge, update and insert statements”
↩︎ Failures surfacing through jobs and the warehouse“this contains all the queries dbt has run.”
↩︎ Checkpoint - 2.
“Unique node identifier, which maps results to nodes in the manifest”
↩︎ Locating the node that broke the run“Note that only executed nodes appear in the run results.”
↩︎ Checkpoint - 3.
“continue: Downstream models continue running when the model fails, instead of being skipped.”
↩︎ How a failure propagates downstream“skip_children (default): All downstream models are skipped when the model fails.”
↩︎ Key concept“the overall run still fails even if all downstream models succeed.”
↩︎ Exam trap 1“the overall run still fails even if all downstream models succeed.”
↩︎ Prediction“The --fail-fast flag takes precedence over on_error: continue.”
↩︎ Checkpoint - 4.
“If other models are in-progress when the first failure happens, then dbt will terminate the connections for these still-running models.”
↩︎ How a failure propagates downstream“fail_fast is a global flag, so it stops on both run errors and test errors”
↩︎ Exam trap 2 - 5.
“Placed after a model/resource — Includes the resource itself and all its descendants (downstream dependencies).”
↩︎ Rerunning only the broken part of the DAG - 6.
“For smarter reruns, use the result:<status> selector instead of manually overriding dbt commands with the models in scope.”
↩︎ Rerunning only the broken part of the DAG - 7.https://docs.getdbt.com/reference/commands/retryOfficial docs
“In each scenario dbt retry picks up from the error rather than running all of the upstream dependencies again.”
↩︎ Rerunning only the broken part of the DAG“retry won't run anything since there are no recorded nodes to retry from.”
↩︎ Exam trap 3 - 8.https://docs.getdbt.com/faqs/Runs/failed-prod-runOfficial docs
“Then, debug these runs the same way you would debug any runs in development.”
↩︎ Failures surfacing through jobs and the warehouse - 9.https://docs.getdbt.com/docs/deploy/retry-jobsOfficial docs
“The modal lists the run steps that will be invoked: the failed step and any skipped steps.”
↩︎ Failures surfacing through jobs and the warehouse“Select Deploy from the top navigation bar and choose Run History.”
↩︎ Checkpoint