Build1 publisher3 min readPublished
Deleting one Parquet file from your own bucket corrupts Snowflake's Iceberg catalog
A dev.to post argues that Snowflake-managed Iceberg tables let you keep the managed engine while owning the Parquet in your own bucket. The same post says the catalog stays Snowflake's, and that 50,000-inserts-per-second ingest should not move.
The Engineer · Build desk
What happened
- A dev.to post argues that Snowflake-managed Iceberg tables keep the managed service while the actual Parquet data files sit in the customer's own cloud storage.
- The author warns that manually deleting a Parquet file from that S3 bucket will corrupt Snowflake's catalog, and that ALTER TABLE ... REFRESH is the command for resyncing after out-of-band changes.
- His stated practice keeps high-velocity staging tables in Snowflake's proprietary format and moves only data at rest, such as daily aggregates and cleaned facts, into Iceberg.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability A Trino cluster or a laptop running DuckDB can read the same tables the warehouse writes, so ad hoc analysis and ML feature pulls stop needing a Snowflake warehouse to be running.
- constraint You own the bytes, but any tool that touches the files outside Snowflake creates a metadata repair job, so the bucket needs the same write discipline as a vendor appliance.
- decision Adoption forces a table-by-table sort on write pattern before anything moves, because the streaming path the post excludes is usually the path with the most data in it.
- cost Teams expecting a compute saving should check which line items actually shrink; the operations the post calls a tax are reads by outside tools, and in-warehouse transforms keep buying credits.
Run the DDL in the post and two clauses set the division of labour. `CATALOG = 'SNOWFLAKE'` keeps metadata management, compaction and schema evolution inside Snowflake [8][10]. `EXTERNAL_VOLUME` points at a bucket you own, and the author calls that volume the bridge between Snowflake and your S3 or GCS storage [9]. The Parquet lands in your account. A Trino cluster or a local DuckDB instance pointed at the same bucket reads the manifest files, the snapshot history and the schema evolution without an export job [11].
The dependency moves from the file format to the catalog. The author is explicit about the cost. "If you manually delete a Parquet file from your S3 bucket, Snowflake's catalog will get corrupted," he wrote, and "You cannot treat your underlying S3 storage as a playground" [12][14]. When files have been changed outside the engine, the sync command is `ALTER TABLE ... REFRESH` [13].
The single catalog is the real gain, and it is measured against a failure the author says he has watched. Engineers wire up AWS Glue crawlers and manual Athena partitions; the Snowflake metadata and the Glue catalog drift apart within three weeks, dashboards break, and a Friday afternoon goes into patching manifest files [7].
The opening figure is a claim about someone else's bill. The author wrote that ninety percent of the performance you pay for in Snowflake is really a tax on your inability to move your data [2]. The post asserts it and shows no bill breakdown [21]. For the number to transfer, your credits have to be dominated by the operations the post names as avoidable: cloning a table, unloading to CSV or Parquet, or starting a compute cluster to read a few gigabytes of rows for ML training [5][20]. A nightly transform that runs inside Snowflake sits outside that list. Move cold tables to Iceberg and the storage bill moves with them, while the transform keeps buying credits.
Write patterns set the boundary. At high-frequency streaming ingestion, on the order of 50,000 inserts per second, the post says Snowflake's native format is still the better target, because Iceberg has to generate new manifest files for every transaction and small-batch writes degrade against standard tables [15]. The author's answer is what he calls "Tiered Persistence": high-velocity staging tables stay in the proprietary format, and data at rest such as daily aggregates and cleaned facts moves to Iceberg [16]. Only the Iceberg tier can be read by anything outside Snowflake [19]. The escape hatch the author describes as "just a standard SQL command" covers the tables that have already stopped moving [17][19].
He warns against applying it to a whole estate, too: "I've seen teams try to force this into every table they own. Don't do that," he wrote [22].
What to watch
- Whether Snowflake's own documentation matches the post's USING ICEBERG syntax and the CATALOG and EXTERNAL_VOLUME defaults. The post does not cite it.
- A measured comparison of Iceberg manifest overhead against standard Snowflake tables at the same batch size, which the post claims without measuring.
- Whether an outside engine can write to a Snowflake-managed Iceberg table, or only read the snapshots Snowflake commits.