Iceberg External Volumes and Snowflake Open Catalog (Polaris)

1.External volumes: the storage connection for Iceberg tables

Apache Iceberg tables keep their data and metadata in your own cloud storage, so Snowflake needs a way to reach that storage. For stages, that is a storage integration. For Iceberg tables, it is an external volume: a named, account-level object that stores an IAM entity for a storage location. Snowflake uses that entity to read and write table data, Iceberg metadata and manifest files. An external volume must exist before you can create an Iceberg table, and one volume can serve many tables.

How many volumes you need depends on how you want to organize and secure the data. A single volume can hold the data and metadata of all your tables in subdirectories of one location. You might instead separate them by access: a read-only volume for externally managed Iceberg tables, and a read-write volume for Snowflake-managed tables. Each storage location has these settings when you add it in Snowsight:

To check that Snowflake can authenticate to the storage, verify the volume. In Snowsight, go to Catalog » External data » External volumes and choose … » Verify connection. In SQL, call the system function:

SELECT SYSTEM$VERIFY_EXTERNAL_VOLUME('my_s3_external_volume');

Sources1

2.Default volumes, USAGE and adding storage locations

Instead of naming a volume in every CREATE ICEBERG TABLE, set the EXTERNAL_VOLUME parameter. An account administrator can set it with ALTER ACCOUNT, which makes it the default for every Iceberg table in the account. Users can override it on a database or schema, and the narrowest setting wins: schema, then database, then account. Setting it on a database or schema requires the usual ALTER privileges plus USAGE on the external volume. USAGE also lets a role reference the volume and view its details. The owner can grant it in Snowsight with + Privilege.

ALTER DATABASE my_database_1
  SET  ?  = 'my_s3_vol';

With a database default in place, a table that uses Snowflake as its catalog needs only a BASE_LOCATION:

CREATE ICEBERG TABLE iceberg_reviews_table (
  id STRING,
  product_name STRING,
  product_id STRING,
  reviewer_name STRING,
  review_date DATE,
  review STRING
)
CATALOG = 'SNOWFLAKE'
BASE_LOCATION = 'my/product_reviews/';

To add another storage location to an existing volume, use the ADD STORAGE_LOCATION parameter of ALTER EXTERNAL VOLUME. In Snowsight, a role with OWNERSHIP on the volume can choose … » Add storage location.

Sources1

3.Snowflake Open Catalog: catalogs, namespaces and storage

The exam guide calls it the Polaris Catalog, and Snowflake's product name is Snowflake Open Catalog: a managed service for Apache Polaris, built on the open Iceberg REST protocol. It gives REST-compatible engines centralized, secure read and write access to Iceberg tables. Note the status: customers without an existing Open Catalog account cannot sign up, and new customers are directed to Snowflake Horizon Catalog. Existing customers can keep using it and create more Open Catalog accounts.

An Open Catalog account holds one or more catalogs. Each catalog stores the current metadata pointer for its tables and updates it atomically. A catalog is one of two types. An internal catalog is managed by Open Catalog, and both third-party engines and Snowflake can read and write its tables. An external catalog is managed by another provider; currently only Snowflake is offered. Its tables are synced into Open Catalog and are read-only there. Namespaces, which can be nested, group the tables inside a catalog.

When you create a catalog you also create its storage configuration. For S3, that means a default base location, the allowed locations, an S3 role ARN and an optional external ID. Azure needs a tenant ID. Open Catalog generates an IAM entity that the cloud provider is told to trust, which is the same pattern as a storage integration. Access privileges are only enforced correctly if each directory holds one table's files and the directory hierarchy matches the namespace hierarchy. Snowflake-managed tables in an external catalog can overlap directories, so give each one a unique BASE_LOCATION. Dropping a table without purging it leaves its data in cloud storage. For that reason, never create a new table with the same name and location as the dropped one.

Sources2

4.Connecting engines: service principals and roles

Engines connect to Open Catalog through service principals. Each one holds credentials, a Client ID and Client Secret pair generated by Open Catalog. A service connection represents an engine such as Spark, Flink or Trino. When an administrator creates one, they give its service principal a new or existing principal role. Privileges are granted to catalog roles, catalog roles are granted to principal roles, and through them the service principal gets its access. A new principal role starts with no privileges until catalog roles are granted to it. Snowflake itself reads Open Catalog tables through a catalog integration, by creating an externally managed Iceberg table.

Sources2

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Changing a database's EXTERNAL_VOLUME moves that database's existing Iceberg tables to the new volume.Why is that wrong?

   The change only applies to tables created afterwards. Existing tables keep the volume they were created with.

   Covered in Default volumes, USAGE and adding storage locations

2. 2.Any new Snowflake customer can sign up for Snowflake Open Catalog to get a managed Polaris catalog.Why is that wrong?

   First-time sign-ups are closed. New customers are directed to Snowflake Horizon Catalog, and only existing Open Catalog customers can create more Open Catalog accounts.

   Covered in Snowflake Open Catalog: catalogs, namespaces and storage