RustFS puts an Iceberg REST catalog on the same port as its S3 API
RustFS embeds an Apache-2.0 Iceberg REST catalog in its object store and serves it on the S3 port, according to a dev.to walkthrough. Teams can drop a separate metastore, but catalog metadata then shares a failure domain with the data.
The Engineer · Build desk
S3 and the Iceberg REST catalog served on one port; catalog metadata has no separate failure domain; the transaction boundary is one table; stale concurrent writers are rejected, duplicates absorbed; Spark, PyIceberg and DuckDB are documented as connecting without a vendor plugin.
- capability Self-hosted lakehouses One process on one port serves S3 and the Iceberg REST catalog, no separate Hive Metastore or Glue, claim 1
- exposure Catalog metadata No separate metadata failure domain; the RustFS backup story is also the catalog backup story, claim 6
- constraint Multi-table transactions Transaction boundary is one table; cross-table consistency needs app logic or eventual consistency, claim 9
- capability Concurrent writers Compare-and-swap version checks reject stale writers; idempotency keys absorb duplicates, claim 8
- capability Spark, PyIceberg and DuckDB Documented as connecting to the RustFS catalog without a vendor plugin, claim 4
| Who | How | Kind | Claim |
|---|---|---|---|
| Self-hosted lakehouses | One process on one port serves S3 and the Iceberg REST catalog, no separate Hive Metastore or Glue | capability | 1 |
| Catalog metadata | No separate metadata failure domain; the RustFS backup story is also the catalog backup story | exposure | 6 |
| Multi-table transactions | Transaction boundary is one table; cross-table consistency needs app logic or eventual consistency | constraint | 9 |
| Concurrent writers | Compare-and-swap version checks reject stale writers; idempotency keys absorb duplicates | capability | 8 |
| Spark, PyIceberg and DuckDB | Documented as connecting to the RustFS catalog without a vendor plugin | capability | 4 |
What happened
- Apache Spark, PyIceberg and DuckDB are documented as connecting with no vendor plugin, and enabling a table bucket does not require registering existing Parquet files first.
- Table creation and snapshot commits go through the REST catalog, while the compute engine writes Parquet files straight into the bucket over the S3 API with no intermediate layer.
- Enabling the catalog takes a dedicated bucket and one signed PUT with an empty body to /iceberg/v1/buckets/my-bucket.
- Commits get two protections. A compare-and-swap check on the table version turns away a concurrent writer working from a stale version, and idempotency keys make a duplicate submission harmless.
Why it matters
- exposure A damaged or lost bucket now takes the catalog with it, because backup and durability for the data are the only backup and durability the catalog has.
- constraint Atomicity stops at one table. A schema change on one table plus a data load into another falls to application logic or eventual consistency.
- decision The only run described is one author's single-node test on RustFS 1.0.1, so a team replacing a live catalog has to supply its own multi-node and write-contention testing.
Two guards sit between the S3 API and the catalog state. Pointers live under a reserved internal prefix, so an ordinary S3 request cannot create, overwrite or delete catalog metadata. The write-up says a stray aws s3 cp will not corrupt table state [7].
That protects against careless clients. It does nothing for the bucket itself. The Iceberg metadata JSON and manifests are written to the same bucket as the data [5], and the author states the consequence: there is no separate metadata failure domain [6]. With Hive Metastore, Glue, Nessie or Polaris beside the store [3], the catalog has its own state to lose and its own recovery. Here one backup covers both, and no independent metastore exists to fall back on [6]. In our view that is a fair price for one fewer service to run [1], provided the store's own durability is already trusted.
Whether an existing job moves over unchanged depends on how many tables it commits. Single-table jobs fit the boundary [9]. A job that loads a fact table and then updates its dimension table makes two commits, and each is atomic only by itself [9].
Setup has one sharp edge. Both the enable call and the status check return HTTP 200 with enabled: true and catalog-type: iceberg-rest [11]. The enable call signs an empty body, so it carries the well-known empty-body SHA-256 in the x-amz-content-sha256 header. Reuse that header on the namespace POST, which carries JSON, and the server rejects it with Sha256Mismatch and HTTP 400. Body-carrying requests need UNSIGNED-PAYLOAD or the real body hash [13]. Signing bugs tend to announce themselves this way.
With the header corrected, namespace creation returned {"namespace":["my-namespace"],"properties":{}} [15].
What to watch
- Whether RustFS or independent users publish multi-node results for the catalog, including commit behavior when several writers target one table.
- Whether RustFS adds multi-table commits, which would remove the per-table limit on fact-and-dimension jobs.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence50
- Adoption
- Insufficient
- Hype gap−10
- Incentives
- Insufficient
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
RustFS S3 Tables is an Apache Iceberg REST Catalog built into the RustFS object storage kernel, so one process on one port serves both standard S3 and the Iceberg REST Catalog protocol, removing the separate Hive Metastore or Glue catalog a self-hosted lakehouse normally needs.
- [2]
RustFS S3 Tables is Apache 2.0 with no separate commercial edition.
- [3]
A typical self-hosted lakehouse pairs the object store with a catalog such as Hive Metastore, AWS Glue, Nessie, or Polaris.
- [4]
Apache Spark, PyIceberg and DuckDB are documented as connecting to the RustFS catalog without a vendor plugin, and enabling a table bucket does not require registering existing Parquet files first.
- [5]
The catalog's metadata lives inside RustFS itself, as Iceberg metadata JSON and manifests written to a reserved prefix on the same bucket.
- [6]
Because catalog metadata lives in RustFS, there is no separate metadata failure domain; the RustFS durability and backup story is also the catalog backup story, and you cannot fall back to an independent metastore.
- [7]
Metadata pointers live under a reserved internal prefix, so ordinary S3 requests cannot create, overwrite or delete catalog metadata; a stray aws s3 cp will not corrupt table state.
- [8]
On commits, the catalog runs compare-and-swap version checks to reject stale concurrent writers and uses idempotency keys to absorb duplicate submissions.
- [9]
The transaction boundary is one table. RustFS does not claim multi-table atomic transactions, so a schema change to one table and a data load to a second cannot commit in one atomic operation; cross-table consistency, such as updating a dimension and a fact table together, must be coordinated in application logic or left eventually consistent.
- [10]
The official walkthrough creates a dedicated bucket, then turns it into a table bucket with a signed PUT to /iceberg/v1/buckets/my-bucket that carries an empty body.
- [11]
Both the enable call and the GET status check return HTTP 200, with enabled: true and catalog-type: iceberg-rest in the response.
- [12]
The author wrote that they confirmed the flow end to end on a single-node RustFS 1.0.1 run.
- [13]
The enable call signs an empty body, so it carries the well-known empty-sha256 x-amz-content-sha256 header; reusing that header on the namespace POST, which carries a JSON body, is rejected with Sha256Mismatch (HTTP 400). Body-carrying requests need x-amz-content-sha256: UNSIGNED-PAYLOAD or the actual body hash.
- [14]
Table creation and snapshot commits travel through the control plane, which is the Iceberg REST Catalog; Parquet data files are written by the compute engine straight into the bucket over the S3 API, with no intermediate layer in the data path.
- [15]
Namespace creation by POST to /iceberg/v1/my-bucket/namespaces with a JSON body returned {"namespace":["my-namespace"],"properties":{}}.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toRustFS S3 Tables: A Built-In Iceberg REST Catalog in an Apache-2.0 Object Store
1 article · October 12, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Object StorageFollow
- Data LakehouseFollow
- Apache IcebergFollow