Build1 publisher2 min readPublished
A single Iceberg tag switches off S3 Tables snapshot expiry for the whole table
AWS schedules compaction, snapshot expiration and orphan cleanup on table buckets itself, and a dev.to teardown finds that managed maintenance halts on any table carrying a user-defined tag or branch.
The Engineer · Build desk
What happened
- Amazon S3 Tables has been generally available across most AWS regions since 2025, merging Iceberg storage and catalog into one managed service built on a bucket type that holds only Iceberg tables.
- Every table bucket exposes an Iceberg REST catalog endpoint that any Iceberg-compatible engine can connect to, and the tables also show up in the Glue Data Catalog through a federated s3tablescatalog.
- AWS runs binpack, sort or z-order compaction in the background with target file sizes configurable between 64 MB and 512 MB, alongside snapshot expiration and orphan cleanup that it schedules itself.
- The dev.to teardown reports that AWS does not document the trigger conditions for auto-compaction, and that a compaction run leaves behind no logs and no before/after file counts.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Iceberg branching and managed snapshot expiration are mutually exclusive on the same table, so a team running write-audit-publish keeps the maintenance job it hoped to delete.
- decision Ingestion code needs explicit retry-and-backoff against background maintenance before it writes to a table bucket, and that is engineering work the zero-ops framing does not account for.
- exposure Streaming pipelines with freshness SLAs are exposed to a maintenance schedule they cannot observe, and the documented route to learning it is an AWS support case.
- contradiction The open REST endpoint weakens the usual lock-in reading of S3 Tables; the cost the post documents is maintenance visibility.
Commit contention is the first thing to test before you point a writer at a table bucket. Maintenance runs continuously in the background, and it can advance table metadata under an in-flight write. The pipeline surfaces that as `CommitFailedException: branch main has changed`, which has to be handled with explicit retry-and-backoff in ingestion code [14]. AWS documents that its own maintenance jobs can fail on "out of date metadata" and get retried [15].
Snapshot management has a harder edge. If any user-defined tag or branch exists on a table, snapshot management fails for that entire table, and nothing is expired or removed until every tag and branch is deleted [16]. Setting the retention properties directly, `history.expire.max-snapshot-age-ms` or `history.expire.min-snapshots-to-keep`, makes snapshot management fail silently [17].
The 10x figure is a claim about someone else's commit path. AWS reports up to 10x higher transactions per second for table buckets than for general-purpose S3 buckets [7]. For that to transfer, the object store has to be what limits your writes. The dev.to post points at the other ceiling: Glue Data Catalog defaults to roughly 10 concurrent `UPDATE TABLE` calls per second per table [8].
Compaction cost fell by up to 90% as of July 2025, according to the post [5], and compaction only touches files in the Frequent Access tier under Intelligent-Tiering, so cold data is left where it was written [9]. Target file size is configurable from 64 MB to 512 MB [4]. At the top of that range the same data volume lands in one eighth as many files as at the bottom [22].
"Credit where it is due: zero-configuration maintenance is a genuine step forward for the ecosystem," the post's author wrote [19]. In my view that holds inside the scope the post gives it: greenfield, AWS-native, running on Athena, EMR and Redshift, where two or three engineers save weeks of infrastructure work [18]. The failure it removes is real, and anyone who has watched a table reach 40,000 small files because nobody scheduled `rewrite_data_files` has seen it [21].
An estate that already runs its own maintenance jobs would be swapping visible maintenance for maintenance AWS does not report on, in exchange for the throughput headroom AWS advertises [11][7]. Cross-region replication is part of the package: read-only replicas across regions and accounts, with full snapshot history preserved [10]. The post's own framing is that production Iceberg estates tend to outgrow what S3 Tables delivers [20].
What to watch
- Whether AWS publishes the auto-compaction trigger conditions or emits per-run compaction logs, which would close the observability gap the post documents.
- Whether snapshot management is fixed to work on tables that carry user-defined tags and branches.
- Whether the Glue Data Catalog default of roughly 10 concurrent UPDATE TABLE calls per second per table is raised.