Build1 publisher2 min readPublished
Choosing Iceberg over Snowflake moves six governance components onto your build list
Keeping the Iceberg table format governance-unaware was a deliberate call. That leaves the catalog as the only thing standing on the path of every read. A dev.to guide lists six pieces an operator has to source above it.
The Engineer · Build desk
What happened
- A dev.to guide describes the Iceberg table format as open, engine-agnostic and deliberately governance-unaware, so any engine can read any table and no single system owns access control.
- Snowflake, BigQuery and Redshift instead ship access control, policy enforcement, retention and audit trails from the vendor, and the guide names the price as data in a proprietary format.
- One reason given for keeping rules out of the metadata is that every policy change would then require metadata migration across potentially thousands of tables, while GDPR, CCPA, DORA and the AI Act keep changing.
- In the guide's model the catalog is the mandatory intermediary: creation, schema alteration, read and snapshot commit route through it, and a denial stops the engine before it sees the metadata.
- Its operational example is a table that passes every access control check but takes 45 minutes to query because it has 200,000 small files.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Six components have to be selected, integrated and kept current by the team that chose the open format, and the same functions arrive as one configurable platform inside a warehouse.
- constraint By the guide's own enforcement logic, a catalog binds only the engines an operator can force through it, so anything that reaches storage on its own credentials sits outside the governance model.
- decision Before paying for the three-layer split, a buyer has to test the substitution it promises: policies in a vocabulary two engines understand, and a catalog able to call more than one decision point.
A rule written into table metadata is only as strong as the reader that honours it. The dev.to post puts that plainly: "A well-behaved Spark job respects the ACL. A misconfigured Trino deployment ignores it. A custom Parquet reader bypasses it entirely." [7] The portability argument runs alongside it. If Iceberg metadata carried ACLs and masking rules, every engine would have to implement the same RBAC structure, the same masking functions and the same row-filter syntax. An engine that implemented only part of the model would either break compatibility or silently bypass security [5].
The same test applies one layer up, and here the guide asserts more than it shows. It treats the catalog as the enforcement mechanism and the policy engine as the rule-authoring system, and says keeping those apart is critical [9]. Credential vending sits in the guide's list of required tools [11]. The post stops short of saying how it keeps a process that already holds bucket credentials from reading the Parquet files directly.
Which brings up the size of the bill. Governance in an Iceberg lakehouse "is not a feature you enable. It is an architecture you assemble," according to the guide [1]. The list of what to assemble runs to catalog-level RBAC, credential vending, the new Read Restrictions spec, policy engines, compliance patterns and an operational governance layer the author says most teams overlook [11]. Six components [14]. Filling that vacuum correctly is, in the guide's words, one of the hardest operational problems in production lakehouses [15].
What the three-layer split buys in return is substitution: Ranger for OPA or Cedar without changing the catalog, Glue to Polaris without rewriting policies, Iceberg v2 to v3 without touching the governance model [10]. For the first of those to be real, policies have to be expressed in a vocabulary both engines understand, and the catalog has to be able to call more than one decision point. The format itself contributes only structural metadata that governance systems can use, including column names for classification and snapshot history for audit, and it never interprets any of it as a rule [16].
For the 45-minute query figure to be yours, you would need a comparable file count, the same engine's planning path and similar storage latency. The post offers it as an illustration of a table that is "technically governed but operationally useless", not as a measurement [12]. The guide names the gap and then names the tool: a control plane like LakeOps, handling autonomous table maintenance, structural health monitoring and compaction across all engines [13].
What to watch
- Detail on the Read Restrictions spec, and whether enforcement lands in the format or stays catalog-side.
- Evidence that a production catalog calls more than one policy engine, the assumption under the Ranger-to-OPA swap.
- Whether auditors accept Iceberg snapshot history as the audit trail for GDPR or DORA obligations.