What you will be able to do
- Explain how a pipe ties a stage to a target table, and which privileges a pipe needs
- Choose between Snowpipe auto-ingest, the Snowpipe REST endpoints and bulk COPY
- Predict how Snowpipe orders, deduplicates and records the files it loads
- Choose between Elastic and Named Channels in Snowpipe Streaming, and place the Kafka connector and Openflow among the ingestion paths
Key concept
Pipe — A pipe is a Snowflake object that wraps a COPY statement. The COPY statement names a stage as the source and a table as the target. Both Snowpipe and Snowpipe Streaming load data through a pipe, so the pipe definition controls what gets loaded and where it goes.
1.Stages and the pipe that reads them
A file-based continuous pipeline begins at a stage. Files land in the stage, and Snowpipe picks them up and loads them as soon as they are available. Snowpipe does not decide what to load on its own. Each load follows the COPY statement defined in a pipe, and that statement names the stage that holds the files and the table that receives the rows. All data types are supported, including semi-structured formats such as JSON and Avro.
The main reason to use Snowpipe instead of scheduled COPY statements is latency. Data arrives in micro-batches and is usually available within minutes, so you don't have to wait for a larger batch on a schedule.
The kind of stage affects the privileges you need. A role that creates a pipe needs USAGE on an external stage, or READ on an internal stage. It also needs SELECT and INSERT on the target table, and CREATE PIPE on the schema. The pipe owner keeps the same stage and table privileges after the pipe is created. A separate role with OPERATE on the pipe can pause or resume it without owning it.
One concrete example of a stage from the sources is the default user stage, named @~. In the Python UDF example, a file is copied from the local file system into @~ with the PUT command from SnowSQL (the example notes that PUT cannot be run through the Snowflake GUI). The SQL API doesn't support the PUT or GET commands either. These sources don't otherwise describe named stages, table stages or directory tables, or how to create a stage, so check the stage documentation before relying on those details.
| Object | Privilege | Notes |
|---|---|---|
| Database | USAGE | |
| Schema | USAGE, CREATE PIPE | |
| Stage in the pipe definition | USAGE | External stages only |
| Stage in the pipe definition | READ | Internal stages only |
| Table in the pipe definition | SELECT, INSERT |
Checkpoint 1 of 6· Check yourself
What does the COPY statement inside a pipe identify?
The pipe's COPY statement names the stage and the target table. Compute comes from Snowflake, and notifications only tell Snowpipe that files have arrived.
“The COPY statement identifies the source location of the data files (i.e., a stage) and a target table.”Source: docs.snowflake.com
2.Auto-ingest compared to the REST API
Snowpipe has two ways to learn that new files are in the stage.
With auto-ingest, your cloud storage sends event notifications. Snowpipe polls those notifications from a queue and loads the new files serverlessly, using the parameters in the pipe. Snowflake recommends turning on cloud event filtering to cut cost, event noise and latency. In government regions, event notifications can't be sent to or from other commercial regions.
With the REST endpoints, your own client application decides when files are ready. It calls a public endpoint with the pipe name and a list of file names, and any matching files in the stage are queued for loading. Snowflake-provided compute resources then load data from the queue into the table. These calls require key-pair authentication with a JSON Web Token (JWT).
The two mechanisms differ in how files are discovered, not in how they are loaded. Each pipe has a single queue that sequences files awaiting loading, and loads use Snowflake-supplied compute either way. The ordering rule (older files usually first, no guarantee) is therefore a property of the pipe, not of the detection mechanism. The sources give no latency or cost figure that separates auto-ingest from REST. They say latency depends on file format, file size and COPY complexity, so measure a typical load.
Snowflake account hosts on AWS, Google Cloud and Azure all support both mechanisms against Amazon S3, Google Cloud Storage and the Azure storage services. This means your cloud provider doesn't decide the choice for you. Choose by who knows when a file is ready: the storage service (auto-ingest) or your application (REST).
Both Snowpipe options behave differently from a bulk COPY run on your own warehouse, as the table below shows.
| Aspect | Bulk data load (COPY) | Snowpipe |
|---|---|---|
| Authentication | Security options supported by the client | REST endpoints require key-pair authentication with JWT |
| Load history | Stored in target table metadata for 64 days | Stored in pipe metadata for 14 days |
| Transactions | Always a single transaction | Combined or split into one or more transactions by rows and size |
| Compute | User-specified warehouse | Snowflake-supplied compute resources |
| Billing | Time each virtual warehouse is active | Compute resources used while loading the files |
Checkpoint 2 of 6· Check yourself
An application calls the Snowpipe REST endpoint to submit a list of staged files. What authentication does the call need?
The REST endpoints require key-pair authentication with a JWT signed with RSA. Bulk loading uses whatever security options the client supports.
“Requires key pair authentication with JSON Web Token (JWT).”Source: docs.snowflake.com
Sources1
3.How Snowpipe orders, deduplicates and records loads
Many Snowpipe troubleshooting questions come down to three behaviours.
Ordering. Each pipe has a single queue. Snowpipe adds newly discovered files to it, but several processes pull from that queue. Older files usually load first, but load order isn't guaranteed to match the order files were staged. If a downstream step needs strict order, it can't rely on file arrival order.
Deduplication. The load metadata on each pipe stores the path and name of every loaded file. It skips a file with a name it has already loaded, even if the content has changed. To load corrected data, use a new file name. Because this metadata belongs to the pipe, Snowflake recommends loading any given set of files with either bulk loading or Snowpipe, but not both. Mixing them can reload files and duplicate data.
History and latency. Pipe load history is kept for 14 days. You request it through a REST endpoint, an SQL table function or an ACCOUNT_USAGE view. It isn't returned the way a COPY statement returns its output. Latency is hard to estimate, because file format, file size and COPY complexity all affect it, so Snowflake suggests measuring a typical set of loads. For a good balance of cost and latency, follow the file sizing guidance and stage files about once per minute.
Checkpoint 3 of 6· Check yourself
A team sees duplicate rows after running a manual bulk COPY on files that a pipe had already loaded. Which guidance did they ignore?
Bulk COPY and Snowpipe keep separate load metadata, so using both on the same files can reload them and duplicate data.
“we recommend loading data from a specific set of files using either bulk data loading or Snowpipe but not both.”Source: docs.snowflake.com
Sources1
4.Snowpipe Streaming: rows instead of files
Snowpipe still needs files. Snowpipe Streaming removes that step. Applications stream rows directly into Snowflake tables or Snowflake-managed Iceberg tables, without staging files. Data can be queryable in as little as 5 seconds, and throughput can reach 20 GB/s per table. Both figures depend on the shape of the workload. Compute scales serverlessly, and billing is based on throughput: credits per uncompressed GB ingested.
Rows still go through a pipe. The pipe's COPY syntax can reorder columns, cast types and apply expressions before rows are committed. Rows travel along a channel, and there are two channel modes:
| Mode | Delivery and ordering | Use when |
|---|---|---|
| Elastic Channels | At-least-once, no ordering guarantee; Snowflake manages channels | Most new applications; IoT, telemetry, distributed producers |
| Named Channels | Ordered, exactly-once within each channel, using offset tokens | Sources needing strict ordering, such as Kafka partitions or CDC |
Producers can connect through the Java, Python or Node.js SDKs, the REST API, or the Snowflake Connector for Kafka. The SDKs buffer, batch and compress appends for you. Direct REST clients have to group rows into NDJSON and compress them themselves, so Snowflake recommends an SDK wherever one is available.
The two services complement each other. Use Snowpipe Streaming when data arrives as rows and must be available quickly. Use Snowpipe when your pipeline already writes files to cloud storage and higher latency is acceptable.
Checkpoint 4 of 6· Match them up
Match each Snowpipe Streaming integration to the workload it fits best
Tap a term, then the definition that fits it.
The overview's integration table recommends each path for a different producer environment. REST is for places where an SDK isn't practical.
“Lightweight workloads, IoT devices, and edge deployments.”Source: docs.snowflake.com
Sources3
5.Kafka connector and Openflow connectors
The exam asks you to compare Snowpipe Streaming with the Kafka connector. In these sources, the two aren't presented as rival products. Snowpipe Streaming is the ingestion service, and the Snowflake Connector for Kafka is listed as one of its ingestion paths, alongside the Java, Python and Node.js SDKs and the REST API. The connector is the path for Apache Kafka topic ingestion. A custom SDK or REST producer fits when the source is an application or a device and not a Kafka topic.
The ordering question follows from the channel modes above. Kafka partitions are the documented example of a source that needs strict ordering, so Kafka data maps naturally to Named Channels and their exactly-once offset tokens. These sources don't cover the connector's installation or configuration options in detail.
Openflow is a separate integration service. It is built on Apache NiFi and connects any data source to any destination, handling structured and unstructured data in both batch and streaming modes. Openflow connectors are curated, versioned Apache NiFi flow definitions, so a connector is a packaged NiFi flow rather than a Snowflake pipe. You can also create your own Openflow dataflow using Snowflake and NiFi processors and controller services.
Openflow has two deployment types. With Openflow - Snowflake Deployment, the flows run on Snowpark Container Services and use Snowflake's security model natively. With Bring Your Own Cloud (BYOC), the data plane runs in your own cloud environment while Snowflake manages the overall service and control plane. Both are available in gen 1 and gen 2, and any new deployment is gen 2.
The sources say to use Openflow when you want to fetch data from any source and put it in any destination with minimal management. Its use cases include CDC replication of database tables, real-time events from services such as Apache Kafka, SaaS data such as LinkedIn Ads, and unstructured content from Google Drive or Box. The catalog includes an Openflow Connector for Kafka and one for Amazon Kinesis Data Streams, a Google BigQuery connector that replicates tables with incremental change capture, and SaaS connectors such as HubSpot, Jira Cloud and Google Sheets.
To place the options: Snowpipe suits files that already sit in cloud storage, Snowpipe Streaming suits rows from applications, devices or Kafka, and Openflow suits connector-based integration from many source systems. Openflow connectors can themselves write through Snowpipe Streaming. The Salesforce Bulk API connector, for example, uses Snowpipe Streaming v2 for its initial load.
Checkpoint 5 of 6· Check yourself
A source system's CDC stream must arrive in Snowflake in order and exactly once per partition. Which Snowpipe Streaming option fits?
Only Named Channels give ordered, exactly-once ingestion within a channel. Elastic Channels are at-least-once and unordered.
“Named Channels provide ordered, exactly-once ingestion within each channel by using offset tokens.”Source: docs.snowflake.com
Checkpoint 6 of 6· Check yourself
What is an Openflow connector?
Openflow is built on Apache NiFi, and its connectors are packaged NiFi flow definitions.
“Openflow connectors are curated, versioned Apache NiFi flow definitions”Source: docs.snowflake.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Snowpipe load history is kept for 64 days, the same as bulk COPY history.Why is that wrong?
The 64 days applies to bulk loads in the target table's metadata. Snowpipe history lives in the pipe's metadata for only 14 days.
Covered in Auto-ingest compared to the REST API
2.Snowpipe loads files in exactly the order they were staged.Why is that wrong?
Several processes pull from the pipe's queue. Older files usually load first, but the order isn't guaranteed.
Covered in How Snowpipe orders, deduplicates and records loads
3.Elastic Channels are the default, so they also guarantee ordered, exactly-once delivery.Why is that wrong?
Elastic Channels are at-least-once with no ordering guarantee. Ordered, exactly-once delivery requires Named Channels.
Covered in Snowpipe Streaming: rows instead of files
4.Openflow always runs entirely inside Snowflake.Why is that wrong?
With Bring Your Own Cloud, the Openflow data plane runs in your own cloud environment while Snowflake manages the control plane.
Covered in Kafka connector and Openflow connectors
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“The COPY statement identifies the source location of the data files (i.e., a stage) and a target table.”
↩︎ Stages and the pipe that reads them“a role that has the following minimum permissions can pause or resume the pipe”
↩︎ Stages and the pipe that reads them“Snowpipe polls the event notifications from a queue.”
↩︎ Auto-ingest compared to the REST API“Your client application calls a public REST endpoint with the name of a pipe object and a list of data filenames.”
↩︎ Auto-ingest compared to the REST API“For each pipe object, Snowflake establishes a single queue to sequence data files awaiting loading.”
↩︎ Auto-ingest compared to the REST API“there is no guarantee that files are loaded in the same order they are staged.”
↩︎ How Snowpipe orders, deduplicates and records loads“A pipe is a named, first-class Snowflake object that contains a COPY statement used by Snowpipe.”
↩︎ Key concept“Stored in the metadata of the pipe for 14 days.”
↩︎ Exam trap 1“there is no guarantee that files are loaded in the same order they are staged.”
↩︎ Exam trap 2“Requires key pair authentication with JSON Web Token (JWT).”
↩︎ Checkpoint“prevents loading files with the same name even if they were later modified (i.e. have a different eTag).”
↩︎ Prediction“we recommend loading data from a specific set of files using either bulk data loading or Snowpipe but not both.”
↩︎ Checkpoint - 2.
“use the PUT command to copy the file from the local file system to the default user stage, named @~”
↩︎ Stages and the pipe that reads them - 3.https://docs.snowflake.com/en/user-guide/snowpipe-streaming/data-load-snowpipe-streaming-overviewOfficial docs
“Throughput-based billing calculated by credits per uncompressed GB of data ingested.”
↩︎ Snowpipe Streaming: rows instead of files“Use Snowpipe when your pipeline already produces files in cloud storage and batch-oriented, higher-latency loading is acceptable.”
↩︎ Snowpipe Streaming: rows instead of files“Apache Kafka topic ingestion using the Snowflake Connector for Kafka”
↩︎ Kafka connector and Openflow connectors“Elastic Channels provide at-least-once delivery without an ordering guarantee.”
↩︎ Exam trap 3“Lightweight workloads, IoT devices, and edge deployments.”
↩︎ Checkpoint“Named Channels provide ordered, exactly-once ingestion within each channel by using offset tokens.”
↩︎ Checkpoint - 4.https://docs.snowflake.com/en/user-guide/data-integration/openflow/connectors/about-openflow-connectorsOfficial docs
“events from Apache Kafka into Snowflake for near real-time analytics”
↩︎ Kafka connector and Openflow connectors“Replicate datasets and tables from Google BigQuery into Snowflake with incremental change capture”
↩︎ Kafka connector and Openflow connectors“Openflow connectors are curated, versioned Apache NiFi flow definitions”
↩︎ Kafka connector and Openflow connectors - 5.
“Use Openflow if you want to fetch data from any source and put it in any destination with minimal management”
↩︎ Kafka connector and Openflow connectors“the Openflow data processing engine, or data plane, runs within your own cloud environment”
↩︎ Kafka connector and Openflow connectors“the Openflow data processing engine, or data plane, runs within your own cloud environment”
↩︎ Exam trap 4 - 6.https://docs.snowflake.com/en/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/aboutOfficial docs
“The connector uses Snowpipe Streaming v2 for the initial load”
↩︎ Kafka connector and Openflow connectors