Build1 publisher3 min readPublished
Databricks adds UNREGISTER to Iceberg's REST catalog spec so a table can leave its catalog intact
Databricks says it added UNREGISTER to the Apache Iceberg REST catalog spec, so a table can change catalogs with zero files copied. A team can only use it if the catalog it is leaving implements the call, and the post demonstrates it on Unity Catalog.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- Iceberg catalogs do not talk to each other, so REGISTER on its own leaves the old catalog active and two catalogs each believing they own the table.
- DROP TABLE cannot be used to end the old catalog's ownership because it deletes the table's data and metadata.
- Databricks says production moves still require stopping writers and repointing jobs, steps it plans to cover in a follow-up post.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost A catalog move becomes a metadata handoff plus a writer pause, so the size of the table no longer sets the data-copy bill for switching catalogs.
- exposure A REGISTER that fails after UNREGISTER leaves the table in no catalog, so any migration script has to persist the returned pointer before making the second call.
- decision Until Databricks publishes its procedure, each team moving tables has to design its own writer freeze and job cutover around the two calls.
The catalog matters because it owns the commit. An engine reading an Iceberg table first asks the catalog to load it. It gets back the current metadata and the data location, then reads Parquet straight from object storage [4]. The catalog holds the authoritative latest state, and it keeps two writers from silently overwriting each other [5].
REGISTER handled the arrival. It hands a metadata location to a new catalog, and that catalog attaches the table [3]. Iceberg catalogs do not communicate, so the old catalog keeps its entry and stays active [6]. Databricks wrote that in that state "neither will throw an error, but the first write operation will fork the table, leading to inconsistent query results and silent data loss" [7]. According to Databricks, the ecosystem had no standard way to end a catalog's management of a table [14]. DROP TABLE does end it, by deleting the table's data and metadata [8].
UNREGISTER is a small call. It is an empty POST to the table resource: `POST <uc-iceberg-rest-base>/v1/{prefix}/namespaces/{namespace}/tables/{table}/unregister` [10]. It removes the catalog entry without touching the data files and returns the table's latest metadata location [9]. Returning the pointer from the same call that ends ownership is good protocol design. The operator gets the location from the catalog giving up the table, at the moment it gives it up. The catalogs still never need to talk to each other [6].
The post overstates its guarantee by one word. Databricks says the unregister, receive, register sequence leaves the table with "exactly one managing catalog at all times" [11]. Between the UNREGISTER response and a successful REGISTER, the table has no managing catalog [1]. What the sequence does guarantee is at most one owner. That is enough to prevent a fork [1].
Writers are the part Databricks has not yet published. It says a production migration still means safely stopping writers and repointing jobs, and it has left those steps to a follow-up post [12]. On a table with continuous ingestion, stopping writers means writes pause for as long as the handoff takes [3].
Whether the endpoint is in the spec and whether a given catalog implements it are separate questions. Databricks says it contributed the endpoint to the Iceberg REST specification [2]. Its example path uses a Unity Catalog base URL [10]. A table can leave a catalog this way only if that catalog implements UNREGISTER. The receiving catalog needs only REGISTER, which the spec already defined [4]. The post does not say which Iceberg release carries the endpoint or which other catalogs have implemented it. Databricks closes by calling Unity Catalog "the most open lakehouse for managing your data" [13], a superlative from the vendor whose catalog now documents its own exit.
What to watch
- Databricks' promised follow-up on stopping writers and repointing jobs, which will show how long the write pause runs for a table under continuous ingestion.
- Which Iceberg release carries UNREGISTER in the REST spec, and whether REST catalogs other than Unity Catalog implement it.
- Whether migration tooling wraps the two calls with automatic re-registration when REGISTER fails after UNREGISTER succeeds.