dbt webhooks for event-driven job integrations

1.Why push events instead of polling for run status

A webhook lets dbt tell another system that a job did something, at the moment it happens. You don't need a script that asks the API every minute whether the nightly run has finished. Instead, dbt makes an HTTP POST with a JSON body to an endpoint you register. Because the server pushes the call instead of the client pulling it, webhooks are also called push APIs or reverse APIs.

The receiving side can be anything that accepts an HTTP request: a serverless function, a Zapier trigger, or your own small app. That is where the value comes from. Common uses are posting a Slack or Microsoft Teams message, opening a PagerDuty incident when a job fails, writing one Datadog log entry per node that ran, and refreshing a Mode report as soon as fresh data lands. Each of these reacts to a job event, and dbt itself does not provide that reaction.

Built-in [job notifications](lesson:6.1) cover a narrower need: a status alert sent to a person or a channel. If all you want in Teams is "job succeeded" or "job failed", sending a notification to the channel's email address does the job with no code. Choose a webhook when the receiver has to process the event: read the run ID, call back into the dbt Admin API for logs, filter on status, or start work in a third-party tool.

Sources123

2.Creating a webhook subscription

A webhook subscription lives at the account level and is set up under Account settings. Before you start, check four prerequisites:

- The account is on the Starter or Enterprise-tier plan. - On Enterprise, you (or the API service token) hold the Account Admin, Admin, or Developer permission set. On Starter, UI access requires a Developer license. - The account runs on a multi-tenant or AWS single-tenant deployment. - The destination system accepts Authorization headers.

You can create a webhook in the dbt UI or through the dbt API, and both use the same fields. Events lets you subscribe to more than one event. Jobs decides which jobs fire the webhook, and its default catches many people out: if you leave Jobs empty, the webhook fires for every job in the account. For a Mode report refresh, that means every CI and development job would refresh the dashboard too. List only the jobs whose completion should trigger the downstream action.

When you click Save, dbt shows a secret token for the webhook. Store it straight away in your receiver's secret store, because it is the key your endpoint uses to prove that a request really came from dbt.

Sources14

3.Choosing between started, completed and errored events

dbt offers three job events, and the important difference is timing, not just meaning. job.run.started fires when a run begins. job.run.completed fires when a run ends, whether it succeeded or failed. job.run.errored fires only when a run errors, so it is a subset of job.run.completed. If you subscribe to both completed and errored, a failed run produces two deliveries, and a Teams channel gets the same failure message twice.

job.run.completed waits until the run's metadata and artifacts have been ingested, meaning dbt has processed that record of the run so its APIs can serve it. Two APIs matter here. The Admin API gives access to things like the logs from a run, and the Discovery API gives model-by-model statuses and other per-node information. Until ingestion finishes, calls to them for that run may come back empty, which is exactly the gap job.run.errored does not wait out.

Receivers that need that detail therefore subscribe to Run completed. Three example receivers show the pattern, each calling back into dbt after the event arrives. A Zapier-based Slack integration extracts run results from the Admin API and posts a summary with error details. A small PagerDuty app checks for failed nodes and creates an alarm through the PagerDuty Events API. A Datadog app writes a log entry per node containing what the Discovery API reports about it. Each branches on runStatus or runStatusCode from the payload. Choose job.run.errored only when speed matters more than detail. If you do, write the receiver to cope with API calls that return nothing for a short while after the event.

Cancelled runs have no webhook event at all. If your on-call channel needs to hear about cancellations, keep the native Slack job notification installed alongside the webhook integration, so it adds to that alert rather than replacing it. The payload below is what a completed run sends. The data object carries the job, run, environment and project identifiers that a receiver needs to call back into dbt.

"eventType": "job.run.completed",
  "webhookName": "test",
  "data": {
    "jobId": "123",
    "jobName": "Daily Job (dbt build)",
    "runId": "12345",
    "environmentId": "1234",
    "environmentName": "Production",
    "dbtVersion": "1.0.0",
    "projectName": "Snowflake Github Demo",
    "projectId": "167194",
    "runStatus": "Success",
    "runStatusCode": 10,
    "runStatusMessage": "None",
    "runReason": "Kicked off from the UI by test@test.com",
    "runStartedAt": "2023-01-31T19:28:07Z",
    "runFinishedAt": "2023-01-31T19:29:32Z"
  }
}

Sources156

4.Verifying that a webhook came from dbt

An endpoint URL is just a public address, so anyone who learns it can POST a fake "run succeeded" to it. Imagine a Mode refresh or a PagerDuty resolve being triggered by a spoofed request. dbt defends against this by signing every delivery. The Authorization header carries an HMAC SHA256 hash of the request body, keyed with the webhook's secret token. Your receiver recomputes that hash from the raw body and compares it with the header.

auth_header = request.headers.get('authorization', None)
app_secret = os.environ['MY_DBT_CLOUD_AUTH_TOKEN'].encode('utf-8')
signature = hmac.new(app_secret, request_body, hashlib.sha256).hexdigest()
return signature == auth_header

Hash the raw body exactly as it arrived. If you parse the JSON and re-serialize it before hashing, the bytes change and the signature will never match. Zapier, a no-code automation service that can receive webhooks, offers two trigger types for this: Catch Hook and Catch Raw Hook. Choose Catch Raw Hook, because only the raw variant gives the code step the untouched body and the Authorization header. Catch Hook is acceptable only if you don't intend to validate authenticity, which is not recommended. Because the signature travels in that header, the destination must accept Authorization headers at all. You can confirm this before wiring anything up with a simple curl request:

curl -H 'Authorization: 123' -X POST https://<your-webhook-endpoint>

Request inspectors such as RequestBin and Requestly show the headers and body of each test delivery, which helps when a signature check keeps failing. One sequencing detail applies to receivers you host yourself, such as the Python apps on fly.io that dbt's PagerDuty and Datadog guides walk through: store the webhook secret in the app before you click Test Endpoint, the button on the webhook's settings in dbt that sends a test event to your endpoint. Until the app has the key, every test delivery fails the signature check.

auth_header = request.headers.get('authorization', None)
app_secret = os.environ['MY_DBT_CLOUD_AUTH_TOKEN'].encode('utf-8')
signature = hmac.new(app_secret, request_body, hashlib. ? ).hexdigest()
return signature == auth_header

Sources165

5.Retries, timeouts and archived webhooks

Delivery is at-least-best-effort, and the receiver has to be built with that in mind. dbt retries each event five times. Each webhook keeps a delivery log for 7 days in its own Recent Deliveries section, which shows at a glance whether each delivery succeeded or failed. That log is the first place to look when an integration goes quiet.

The 10-second timeout catches slow receivers. When the endpoint does its work before responding, it can finish happily and log success while dbt marks the delivery as failed. Both sides then disagree about what happened, and retries may deliver the same event again. Keep the work done before the response short, or make the downstream action safe to repeat. One way is to key the downstream action on the run ID in the payload: a PagerDuty receiver that deduplicates its events per run ID turns a repeated delivery for the same run into the same event rather than a second alarm.

A webhook that keeps failing doesn't fail forever. dbt archives it automatically after 1,000 consecutive failed deliveries or 7 consecutive days of failed deliveries, whichever comes first. On Account settings → Webhooks, a dismissible banner warns that a subscription is archived, and the status column shows an Archived badge instead of an HTTP status. To bring it back:

- In the dbt platform UI, change the webhook's endpoint URL. A URL change reactivates the webhook automatically. - Through the REST API, send a PUT to Update a webhook with active set to true, or with a new client_url. - With the Terraform provider, set active = true and update client_url in the webhook resource.

Sources15

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Leaving the Jobs field empty means the webhook fires for no jobs until you add some.Why is that wrong?

   An empty Jobs field subscribes the webhook to every job in the account, CI and development jobs included.

   Covered in Creating a webhook subscription

2. 2.Subscribing to both Run completed and Run errored gives the most reliable failure alerting.Why is that wrong?

   Errored is a subset of completed, so a failed run sends two deliveries and the channel receives duplicate messages. Pick one of the two.

   Covered in Choosing between started, completed and errored events

3. 3.A webhook-based Slack integration can fully replace the native Slack notification, including alerts for cancelled runs.Why is that wrong?

   There is no webhook event for a cancelled run, so the native notification is still needed to hear about cancellations.

   Covered in Choosing between started, completed and errored events

4. 4.If the receiver logs a 200 response, dbt also records the delivery as successful.Why is that wrong?

   dbt gives up after 10 seconds. A receiver that responds later logs a success while dbt counts the delivery as a failure.

   Covered in Retries, timeouts and archived webhooks