Build1 publisher3 min readPublished
Apache Iceberg moved lock-in up to the catalog layer, an AWS Community Builder argues
AWS Community Builder Aki argues Apache Iceberg made data portable while permissions, auditing, lineage and sharing stay tied to each catalog. The spec leaves the atomic commit to each catalog, so the catalog is now the choice that is hard to reverse.
The Engineer · Build desk
What happened
- Aki, an AWS Community Builder, wrote that what Apache Iceberg achieved is data portability, with Parquet data and table metadata kept in open specifications.
- In his account, governance (permissions, auditing, lineage and sharing) does not travel with the data the way the files do.
- The Iceberg Table Spec requires the metadata pointer swap to be atomic but does not standardise how, leaving it to each catalog implementation.
- Under the REST Catalog Protocol, clients send requirements and updates, and the server validates them and applies the commit itself.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Picking a catalog becomes the long-lived choice in an Iceberg platform, because tables can move while grants, audit trails and lineage have to be set up again on the other side.
- constraint Engines share an Iceberg table by sharing its catalog, so every engine added to an estate has to work with whichever catalog holds the pointer.
- exposure With REST, the catalog server validates and applies every commit, putting the catalog operator on the write path of every engine that uses it.
A write under the Iceberg Table Spec happens in two steps. First the writer builds a new metadata file, assuming the current version will not change before it commits. Then it swaps the table's pointer from the base version to the new one [9]. If another writer got there first, the compare-and-swap catches the stale base and the commit is treated as a conflict [9]. Schemas, snapshots and partition specs all live inside metadata.json. The snapshot list behind time travel is not the catalog's job either [10]. Those files sit in open specifications next to the Parquet data in storage such as S3 [6].
On paper the catalog has a short job list. It creates, drops and renames tables. Its most important job is tracking which metadata file is current and handing it over when a table loads [7]. That pointer is the piece of table state that lives outside the open files. The spec says the atomic operation behind the swap depends on how tables are tracked [11]. Two catalogs can both follow the spec and still keep the pointer in completely different places.
The REST Catalog Protocol exists so more languages and engines can take part [12]. It also moves the swap onto a server, so the client never touches storage to commit. The server checks each requirement against current metadata before it applies anything [13]. This is good engineering. Conflict detection happens in one place instead of being reimplemented in every client library. The Iceberg docs describe sharing one catalog as the way multiple engines share a common data layer [8].
Aki builds on AWS, Databricks, Snowflake and Microsoft Fabric [4]. He describes the old arrangement this way: in traditional warehouses and lakes, the file format, metadata management, transactions, partitions and access APIs were all closed inside the service. Leaving meant extracting and converting all of the data [5]. He wrote that "what Iceberg achieved is data portability" [1]. His verdict on the rest: "My current view is that lock-in hasn't disappeared; it has simply moved up to a higher layer," he wrote [3].
The spec passages he cites deal with commits. They do not cover grants or audit logs. The governance half of the argument therefore rests on Aki's working experience across those platforms [4]. He also wrote, "By the way, I'm a big fan of Unity Catalog!" [14]. More posts about vendors could stand to include a disclosure like that.
I think the claim holds, for a reason the spec itself gives. For governance to move as easily as data, it would have to live in files the spec defines. The contents the post places in metadata.json are schemas, snapshots, partition specs and manifest lists [10]. Permissions are not on that list. I'd expect a move between catalogs to carry the tables across cleanly and leave the grants to be recreated on the other side.
What to watch
- A common REST Catalog Protocol format for permissions or audit records would put Aki's claim to a direct test.
- A documented migration of Iceberg tables between two catalogs that reports what happened to grants and lineage would show whether governance moved with the data.