Overview
A typical request: a team that runs its analytics on Snowflake needs another engine, such as Spark for data science, to use some of the same tables. AceMQ works with Snowflake specialists to choose which tables become Snowflake-managed Iceberg tables, connect the second engine through the Horizon Iceberg REST Catalog, and write the access rules down. It is a scope-and-process engagement, not a migration.
Challenge
Snowflake's documentation limits what an external engine can do. Native tables cannot be queried that way, writes from the external engine have restrictions, network and encryption settings need specific configuration, and egress is billed when the engine reads from another region or cloud. Teams that skip these checks find them in production: a table the second engine cannot see, a write that fails, or a data-protection policy nobody confirmed would still apply.
Environment
Snowflake on AWS with Iceberg tables on cloud storage the team manages, one or more external engines, and existing masking or row access policies on the data.
Approach
AceMQ classifies the tables the second engine needs as native, Snowflake-managed Iceberg or externally managed, and proposes which to convert. It then designs the access path through the Horizon Iceberg REST Catalog API with vended credentials, checks the documented read and write limits against what the second engine must do, and tests the data-protection policies from the external side. The rules are written as a short access standard the team owns.
Solution
- Table classification and a conversion shortlist for tables that other engines must read
- Connection design for the external engine through the Horizon Iceberg REST Catalog API with vended credentials
- Read and write test plan against Snowflake's documented limits, including deletes, table creation and shared tables
- Check that masking and row access policies apply when the external engine reads, with the results recorded
- Private connectivity, encryption key and storage-access settings reviewed for the cloud in use
- Cost note on egress for cross-region or cross-cloud reads, and an access standard covering roles, owners and review
Outcome
The team ends with a classified table list, a working and documented access path for the second engine, test results for the limits that matter to them and a written access standard. Native tables stay in Snowflake; the tables that move to Iceberg are chosen deliberately.
Technologies
SnowflakeApache Iceberg
Apache Spark
dbt
Terraform
Airflow
S3
Kubernetes
Kafka
PostgreSQL
