Iceberg 1.11.0 puts Spark, Flink and Hive file access behind one format registry
Iceberg 1.11.0 ships a File Format API that sends Spark, Flink, Hive and plain-Java reads and writes through one registry of format models. A dev.to walkthrough says the file format list stays fixed, so Vortex and Lance still need a change inside Iceberg.
The Engineer · Build desk
Spark, Flink, Hive and plain Java reuse iceberg-core readers, so the registry reaches them. Trino uses Iceberg core for metadata, own readers for data. DuckDB, DataFusion, PyIceberg and Polars use Arrow readers; Lance and Vortex ship one. New formats need a change in Iceberg.
- capability Spark, Flink and Hive Link the iceberg-core jar and reuse its readers, so the registry reaches them, claim 7
- decision Trino Uses Iceberg's core for metadata and its own Parquet and ORC readers for data, claim 9
- exposure Arrow-reading engines DuckDB, DataFusion via iceberg-rust, PyIceberg and Polars use their own Arrow readers, claim 8
- capability Lance and Vortex Both ship an Arrow reader, which is all an Arrow engine needs for a new format, claim 10
- constraint New file formats The format list is fixed; a new format remains a change inside the Iceberg project, claim 14
| Who | How | Kind | Claim |
|---|---|---|---|
| Spark, Flink and Hive | Link the iceberg-core jar and reuse its readers, so the registry reaches them | capability | 7 |
| Trino | Uses Iceberg's core for metadata and its own Parquet and ORC readers for data | decision | 9 |
| Arrow-reading engines | DuckDB, DataFusion via iceberg-rust, PyIceberg and Polars use their own Arrow readers | exposure | 8 |
| Lance and Vortex | Both ship an Arrow reader, which is all an Arrow engine needs for a new format | capability | 10 |
| New file formats | The format list is fixed; a new format remains a change inside the Iceberg project | constraint | 14 |
What happened
- Before the API, Spark had SparkParquetReaders, Flink had FlinkParquetReaders and the generic Java API had GenericParquetReaders, each with its own path through read options and pushdown.
- DuckDB, DataFusion through iceberg-rust, PyIceberg and Polars implement the spec with their own readers and read through Arrow, so the Java registry does not reach them.
- Trino is Java too, but it takes metadata from Iceberg core and reads data with its own Parquet and ORC readers.
- Multimodal support is arriving as Parquet column types: FILE shipped in parquet-format 2.14.0, VECTOR is at proposal stage, and both are to be mirrored into the next table spec.
Why it matters
- contradiction The API was first framed as a way to try Vortex and Lance, formats built very differently from Parquet, but the shipped design fixes the format list and opens only the engine side.
- constraint Engines that read through Arrow need only an Arrow reader for a new format, and Lance and Vortex already ship one, so those two formats do not depend on the registry.
- capability A Java engine with its own row type can register a format model and read every Iceberg table through the shared builders.
The registry is a lookup table, and the code is short. The key is a pair of file format and row type. The value is the format model that reads and writes that combination [12]. It sits in iceberg-core [7]. Entries go in through `public static synchronized void register(FormatModel<?, ?> formatModel)`, and the key type is `Pair<FileFormat, Class<?>>` [13]. The excerpt and the grid in the walkthrough come from 1.12.0, not 1.11.0 [13].
The signature shows which side is open. The second half of the key is a Class, the engine's row type, and any engine can register its own [14]. Spark wants an InternalRow, Flink a RowData, the plain Java API an Iceberg Record, and the same Parquet file can be read into any of them [11]. The first half is a FileFormat. According to the walkthrough, that list is fixed, so a new format remains a change inside the Iceberg project [14].
Java needs a registry because it has three row types and no shared memory format [10]. Arrow plays that role in the other engines.
The account here comes from a single dev.to walkthrough. Its description of the builders is plain: every engine now goes through the same ReadBuilder and FileWriterBuilder, with delete-file siblings for the writer [5]. The grid still has one cell per format and row type [11]. The walkthrough does not say how much of a new type's work lands in shared code and how much in each cell.
That split decides whether a v3 type or a shredded variant column reaches Spark and Flink on the same day [3]. Under the old layout the work had to be redone per engine, and the walkthrough says new features tended to arrive elsewhere later or not at all [3]. We think the registry is the right design for three row types with no common memory format. We would not read it as a guarantee that a new type arrives in every engine at once.
What to watch
- Whether the VECTOR type moves from proposal into a released parquet-format version and the next Iceberg table spec.
- Iceberg documentation or release notes showing whether a new v3 type is added once in shared builders or separately in each format model.
- Whether an engine that keeps its own readers today, such as Trino, registers a format model of its own.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap−10
- Incentives
- Insufficient
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Apache Iceberg traditionally exposed its data files through three libraries, one each for Parquet, Avro and ORC, and each engine that spoke Iceberg had its own way of calling them.
- [2]
Spark had SparkParquetReaders, Flink had FlinkParquetReaders and the generic Java API had GenericParquetReaders, and each engine chose its own path through the format's read options, filter pushdown, split handling, encryption and metrics.
- [3]
A feature added to one format for one engine (a new v3 type, a default value, a shredded variant column) tended to arrive for the other engines later or not at all, because the work had to be redone per engine.
- [4]
The File Format API shipped in Iceberg 1.11.0 and was announced in February.
- [5]
The File Format API consists of a single registry of format models, each the pairing of a file format with an engine's in-memory row type, and a single set of builders (ReadBuilder, FileWriterBuilder and its delete-file siblings) that every engine now goes through.
- [6]
The API was originally framed as the way to try out newer formats such as Vortex and Lance, which are designed very differently than Parquet and meant specifically for vector work and multimodal data storage.
- [7]
The File Format API is a Java API. The registry lives in iceberg-core, so it reaches exactly the engines that link that jar and reuse its readers: Spark, Flink, Hive and the plain Java Record path.
- [8]
DuckDB, DataFusion via iceberg-rust, PyIceberg and Polars each implement the Iceberg spec with their own readers and use Arrow readers instead.
- [9]
Trino, which is Java, uses Iceberg's core for metadata and its own Parquet and ORC readers for data.
- [10]
For engines that read into Arrow, the row-type axis collapses to one and a new format just needs an Arrow reader (Lance and Vortex both ship one). Java implementations have three row types and no shared memory format, which is why a registry was needed to play the role Arrow plays elsewhere.
- [11]
Spark wants an InternalRow, Flink a RowData and the plain Java API an Iceberg Record; the same Parquet file can be read into any of them. A format model is one cell in the grid: one format, one row type.
- [12]
In FormatModelRegistry the key is a pair (which file format, which row type) and the value is a format model, the object that knows how to read and write that one combination.
- [13]
The registry's register method is public static synchronized void register(FormatModel<?, ?> formatModel) and builds a key of type Pair<FileFormat, Class<?>>; the code excerpt, and the grid of format models, are given as of 1.12.0.
- [14]
The API makes the engine's row type pluggable over a fixed list of file formats, so any engine can register its own representation and read every Iceberg table through the shared builders, while a new format remains a change inside the Iceberg project.
- [15]
Because the formats are fixed, multimodal support is arriving as new column types defined on the Parquet side (the FILE type shipped in parquet-format 2.14.0, the VECTOR type is in proposal stage) and mirrored into the next table spec.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Columnar File FormatsFollow
- Data LakehouseFollow
- Apache IcebergFollow