Monitoring Snowflake Pipelines: Tasks, Alerts, Dynamic Tables, Streaming and Data Quality

1.The monitoring ladder: snapshot, recent history, long history, events

A continuous pipeline has no single moment when someone watches it run. Tasks fire on schedules, dynamic tables refresh to meet a target lag, and Snowpipe Streaming commits rows from channels. Monitoring it means picking the Snowflake surface that holds the right information for the right length of time.

The choice is clearest for dynamic tables, and the same pattern repeats for other objects. A SHOW command gives a current snapshot with no history. Information Schema table functions give per-run detail for recent activity. Account Usage views keep a long history for trend analysis. An event table combined with an alert is how you get automated failure notifications instead of running queries by hand. Retention is what separates the middle two: a question about the last 30 days cannot be answered from a source that keeps only 7.

Sources1

2.Tasks: run history in Snowsight and SQL

Tasks are scheduled SQL, so the first thing to monitor is whether each run succeeded. Snowsight shows this without any SQL. Go to Catalog » Explorer, open the database and schema, select Tasks, and choose a task. The task page has Graph and Run History tabs. The Graph tab shows the root task, its dependent tasks and any finalizer task. Select a node to see its predecessors, warehouse and owning role. The Run History tab lists past executions, and you can retry failed tasks from Snowsight.

On this per-task page, history is only available if the task has run in the last 7 days, so a task that last ran longer ago shows nothing there. For longer history, use SQL.

There are three TASK_HISTORY surfaces in SQL. The Account Usage view retrieves task history for the last 365 days, with latency of up to 45 minutes. The Organization Usage view covers the accounts in an organization, is available only in the organization account, and can lag by up to 180 minutes. The Information Schema TASK_HISTORY table function returns executions from the past seven days, or the next scheduled execution within the next eight days. It returns at most 10,000 rows, set by RESULT_LIMIT (default 100), so filter by task name, database, schema or a scheduled-time range.

The views record each completed run with a STATE of SUCCEEDED, FAILED, CANCELLED or SKIPPED. A timed-out run always appears as FAILED. The views do not include runs that are still scheduled or executing. For those, query the TASK_HISTORY table function in the Information Schema.

To find out *who* ran a task, use QUERY_HISTORY. A task that does not run as an actual user shows the user as SYSTEM.

Repeated failures also change the task's state. SUSPEND_TASK_AFTER_NUM_FAILURES (default 10) suspends a standalone task, or the root of a task graph, after that many consecutive failed runs. Runs that were skipped or canceled, or that failed because of a system error, do not count toward that number. So if a task stops running without warning, check whether it was auto-suspended before you assume the scheduler is broken.

Tasks can run on serverless compute instead of a warehouse. A task created without a warehouse, like the notification example in the next section, is a serverless task. For a serverless task, the TARGET_COMPLETION_INTERVAL property is used to determine the size of the compute resources. It is optional for serverless tasks and required for serverless triggered tasks. SUSPEND_TASK_AFTER_NUM_FAILURES applies to tasks on either serverless or warehouse compute.

Sources2345678

3.Notifications: pushing task failures to a cloud message service

Checking run history is pull-based: someone has to look. Task notifications are push-based. Snowflake publishes a message when a task errors, or when a task graph finishes successfully, so downstream systems can react without querying Snowflake.

The mechanism is a notification integration object that connects Snowflake to a cloud queue: Amazon SNS, Microsoft Azure Event Grid or Google Pub/Sub. Four rules come up repeatedly:

- Root task only. You set ERROR_INTEGRATION on the root task of a graph, and failures in child tasks go to the root task's integration. - Same cloud. Cross-cloud delivery is not available, so the queue must be on the cloud platform that hosts your account. - Not email or webhook. Those integration types cannot carry task error notifications. - At-least-once delivery. Consumers may receive duplicate messages and must handle them.

Tasks with TASK_AUTO_RETRY_ATTEMPTS above 0 send a notification for every failed run, so one underlying problem can produce several messages. A retried run is visible in task history: its SCHEDULED_FROM value is AUTOMATIC RETRY and ATTEMPT_NUMBER, which is initially one, counts the attempts. Failed runs also count toward SUSPEND_TASK_AFTER_NUM_FAILURES, which suspends the task automatically after that many consecutive failures. To audit what was sent, query the NOTIFICATION_HISTORY table function. The message payload identifies the task and its errors, and it arrives as a string that you must parse into JSON.

CREATE TASK mytask
  SCHEDULE = '5 MINUTE'
   ?  = my_notification_int
  AS
  INSERT INTO mytable(ts) VALUES(CURRENT_TIMESTAMP);

Sources9103

4.Alerts: a condition, an action, and a schedule

Notifications cover task errors. For any other condition you can express in SQL, such as too many failed runs in the last hour, credit consumption rising, or rows breaking a business rule, Snowflake provides alerts. An alert is a schema-level object that combines three things: a condition, an action to take when the condition is met (for example, sending an email or writing to a table), and when to evaluate it.

Alerts come in two kinds:

An alert on new data fits event-driven monitoring. Dynamic table refreshes and task executions write events to the event table, so an alert on new data against that table fires when an error event arrives instead of waiting for the next fixed interval. Iceberg tables cannot be the source, even with change tracking enabled.

An alert needs compute. A serverless alert lets Snowflake size the compute, up to the equivalent of an XXLARGE warehouse, and its usage appears in SERVERLESS_ALERT_HISTORY. Alternatively, you can name a virtual warehouse. For new data that arrives infrequently, choose serverless: with a warehouse, even a simple email action incurs at least one minute of warehouse cost.

To create an alert, a role needs EXECUTE ALERT on the account, which only ACCOUNTADMIN can grant. It also needs either EXECUTE MANAGED ALERT (for serverless) or USAGE on the warehouse, plus USAGE and CREATE ALERT on the schema. An alert on new data also needs SELECT on the source.

Sources11

5.Dynamic tables: freshness and refresh failures

For a dynamic table, monitoring asks two questions: is the table fresh enough, and did the last refresh succeed? Viewing a dynamic table's metadata requires MONITOR or OWNERSHIP on it.

SHOW DYNAMIC TABLES is the only source for refresh_mode, refresh_mode_reason and warehouse. For freshness across many tables at once, use the DYNAMIC_TABLES function:

SELECT
  name,
  database_name,
  schema_name,
  scheduling_state,
  last_completed_refresh_state,
  target_lag_sec,
  time_within_target_lag_ratio,
  maximum_lag_sec
FROM
  TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLES())
ORDER BY
  name;

The key column is time_within_target_lag_ratio. A value below 0.90 means the table misses its freshness target most of the time. When that happens, look at refresh history and consider changing the target lag or the warehouse size.

DYNAMIC_TABLE_REFRESH_HISTORY returns one row per refresh, with its state, error code and message, duration and trigger. One pattern matters for diagnosis: when an upstream dynamic table fails, its downstream tables report UPSTREAM_FAILED with the message "Skipped refreshing because an input dynamic table failed." Fix the first FAILED table in the chain, not the tables that report UPSTREAM_FAILED.

DYNAMIC_TABLE_GRAPH_HISTORY records the dependency graph over time. Each property change, such as a new target lag, adds a row with a new valid_from. The labels differ between commands: SHOW DYNAMIC TABLES reports RUNNING where the graph function reports ACTIVE, and both mean the same scheduling state.

SELECT
  name,
  state,
  state_code,
  state_message,
  data_timestamp
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY(
  NAME_PREFIX => 'MY_DB.MY_SCHEMA',
   ?  => TRUE
))
ORDER BY data_timestamp DESC;

Sources1

6.Snowpipe Streaming: telemetry in the event table

Snowpipe Streaming has two monitoring sides. Server-side, Snowflake writes ingestion telemetry to an event table. Client-side, you watch the SDK with Prometheus and logs. Event-table telemetry requires a pipeline on the high-performance architecture and an active event table. The default is SNOWFLAKE.TELEMETRY.EVENTS, which SNOWFLAKE.EVENTS_VIEWER holders read through EVENTS_VIEW. If both a database-level and an account-level destination are set, the database-level one wins. Check with SHOW PARAMETERS LIKE 'EVENT_TABLE'.

An active event table does not, on its own, collect anything. LOG_EVENT_LEVEL defaults to OFF. At INFO all five event types are recorded, at ERROR only row and channel errors, and at OFF none. Ingestion itself continues normally at any level. Turning collection on does not backfill events for earlier ingestion, and a normal SQL INSERT generates no streaming events. To confirm collection works, you have to send a new batch.

Every query filters on scope:"name" = 'snow.snowpipe.streaming' and record_type = 'EVENT'. It identifies the target by database, schema and table, because snow.pipe.name is not always present. The commit-event query below computes a per-channel error rate:

SELECT
  resource_attributes:"snow.table.name"::STRING AS table_name,
  value:"channel_name"::STRING AS channel_name,
  SUM(value:"row_count"::NUMBER) AS rows_ingested,
  SUM(value:"rows_parsed"::NUMBER) AS rows_parsed,
  SUM(value:"error_count"::NUMBER) AS rows_with_errors,
  SUM(value:"error_count"::NUMBER)
    / NULLIF(SUM(value:"rows_parsed"::NUMBER), 0) AS row_error_rate
FROM <event_table>
WHERE scope:"name"::STRING = 'snow.snowpipe.streaming'
  AND record_type = 'EVENT'
  AND record:"name"::STRING = 'commit'
  AND resource_attributes:"snow.database.name"::STRING = '<database_name>'
  AND resource_attributes:"snow.schema.name"::STRING = '<schema_name>'
  AND resource_attributes:"snow.table.name"::STRING = '<table_name>'
  AND timestamp > DATEADD('minute', -15, CURRENT_TIMESTAMP())
GROUP BY 1, 2
ORDER BY rows_with_errors DESC, rows_ingested DESC;

Two caveats apply. First, uncompressed_bytes can repeat across channels in the same request, so summing it double-counts. Second, total_latency_ms measures only from arrival at Snowflake's ingestion endpoint to commit. It is not end-to-end time from the source system. For Named Channels, use channel status, not event order, to decide where to resume after an interruption.

Sources12

7.Streams: what monitoring a stream can and cannot change

Streams provide change data capture for pipelines. They can be created on standard tables, views, dynamic tables, event tables, external tables and directory tables. When you monitor a stream, what matters is the offset. A stream holds no table data. It stores an offset and uses the source object's version history to return the changes made since that offset, with the metadata columns METADATA$ACTION, METADATA$ISUPDATE and METADATA$ROW_ID.

Any number of queries can read the same change data from a stream, which makes it safe to inspect. The offset moves only when a DML transaction that reads from the stream commits. If that transaction fails, the offset stays put, so the next run picks up the same changes.

Wrapping several statements in BEGIN … COMMIT locks the stream so they all see the same change records. Inside the transaction the stream has repeatable-read isolation.

To skip a backlog without processing it, recreate the stream with CREATE OR REPLACE STREAM, or insert from the stream into a temporary table using a filter such as WHERE 0 = 1.

One failure mode is specific to streams: an incompatible schema change between the offset and the advance can make queries on the stream fail.

Streams also drive tasks. A task can include a WHEN clause with a SYSTEM$STREAM_HAS_DATA condition, which makes the task's run depend on whether the stream holds change data capture (CDC) records. That is the usual way to run a MERGE only when new data exists. The task's CONDITION_TEXT in task history records the WHEN condition it evaluated, and a run started because the stream contained new data shows SCHEDULED_FROM = TRIGGER. If such a task does not run as expected, check that the stream contained CDC records when the task was last scheduled to run. You can query a stream's historical data with an AT | BEFORE clause.

Sources13143

8.Data quality: data metric function results

The earlier surfaces tell you whether the pipeline *ran*. Data metric functions (DMFs) tell you whether the data it produced is *good*. Snowflake includes system DMFs such as FRESHNESS, DUPLICATE_COUNT and BLANK_COUNT, and you can associate them with tables and views.

DMFs run on a schedule set on the object with the DATA_METRIC_SCHEDULE parameter, and every DMF on that table or view follows the same schedule. An interval schedule accepts 5, 15, 30, 60, 720 or 1440 minutes, with a default of 60 MINUTE; a cron expression is also allowed. A third option is 'TRIGGER_ON_CHANGES', which runs the DMF when a DML operation modifies the table, such as inserting a row. It is available for dynamic, external, Iceberg, regular, temporary and transient tables, but you cannot specify it for views. Setting the schedule to an empty string suspends every DMF on that object.

ALTER TABLE hr.tables.empl_info SET DATA_METRIC_SCHEDULE = 'TRIGGER_ON_CHANGES';

You then associate a DMF with specific columns using ADD DATA METRIC FUNCTION:

ALTER TABLE customers ADD DATA METRIC FUNCTION invalid_email_count ON (email);

Scheduled DMF results land in an event table, and there are three ways to read them. The table function returns the same columns as the view, including measurement_time, metric_name and value, but for one table only. Querying the view requires the SNOWFLAKE.DATA_QUALITY_MONITORING_VIEWER or ..._ADMIN application role. When you evaluate results, use the measurement_time column as the basis, because it records when the DMF was evaluated, which can differ from the scheduled time.

DATA_QUALITY_MONITORING_RESULTS(
  REF_ENTITY_NAME => '<string>' ,
  REF_ENTITY_DOMAIN => '<string>'
  )

Snowsight offers a visual surface too. Select the object in Catalog » Explorer, open the Data Quality tab and select Monitoring. The DMFs associated with the object are listed under Quality Dimensions, and a Run Schedule widget shows how often they run. A data quality check is a DMF association with an expectation. When a check fails, select the DMF association to see which expectation was violated, then use View failed records to run a prepopulated query that calls SYSTEM$DATA_METRIC_SCAN.

Separately, automatic data quality monitoring picks popular tables on its own and records anomaly verdicts for metrics such as row count and freshness in AUTOMATIC_DATA_QUALITY_MONITORING_RESULTS. It does not incur compute charges; billing applies only to DMFs you configure explicitly. A data rule that fails is exactly the kind of condition the alerts section described, so DMF results are a natural input for an alert.

Sources151617181920212211

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.You can send task error notifications through an email or webhook notification integration.Why is that wrong?

   Task error notifications support only cloud messaging services (Amazon SNS, Azure Event Grid, Google Pub/Sub) on the cloud that hosts your account.

   Covered in Notifications: pushing task failures to a cloud message service

2. 2.Each child task in a task graph needs its own ERROR_INTEGRATION.Why is that wrong?

   You set it on the root task only, and child task failures are sent to the root task's integration.

   Covered in Notifications: pushing task failures to a cloud message service

3. 3.An active event table means Snowpipe Streaming telemetry is already being collected.Why is that wrong?

   Collection is controlled separately by LOG_EVENT_LEVEL, which defaults to OFF. You must set it to INFO or ERROR on the schema, database or account.

   Covered in Snowpipe Streaming: telemetry in the event table

4. 4.DYNAMIC_TABLE_REFRESH_HISTORY in the Information Schema can answer questions about failures from last month.Why is that wrong?

   The table function keeps 7 days. Older history comes from the Account Usage view, which keeps 365 days.

   Covered in Dynamic tables: freshness and refresh failures