Skip to content

Build1 publisher3 min readPublished

Iceberg's read restrictions hand row filters to an engine the catalog must trust

A new Iceberg REST Catalog addition lets the catalog evaluate policy for one principal and return nine column actions plus row filters for a trusted engine to apply, while the spec leaves establishing that trust to the deployment.

The Engineer · Build desk

Illustration accompanying Iceberg's read restrictions hand row filters to an engine the catalog must trust

What happened

  • According to a Databricks post on open lakehouse governance, the Apache Iceberg community has advanced two additions to the Iceberg REST Catalog: read restrictions and catalog labels.
  • The specification says what a trusted engine must enforce but not how a catalog establishes that trust, and a claim from the client is explicitly not sufficient.
  • Databricks' existing Cross-engine ABAC takes the other route, putting a filtering fleet behind the Iceberg REST scan and plan APIs so data is sanitized before Spark or DuckDB processes it.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint A masking rule that reaches a lookup table, a subquery or a UDF cannot be reduced to the standard vocabulary, so those tables either stay behind catalog-side filtering or ship with weakened semantics.
  • decision Because trust follows the deployment, a team that wants delegated enforcement has to move the query engine somewhere users cannot reach the runtime, or keep filtering inside the catalog.
  • exposure Any engine a catalog wrongly treats as trusted receives the rows and columns the policy was meant to remove, and the spec offers no mechanism to catch that error.
  • cost Adopters carry the unsolved parts themselves: identity propagation, telling the user apart from the engine acting for them, and binding credentials to their intended recipient.

Load a table through the Iceberg REST Catalog and the catalog evaluates the policies that apply to the requesting principal and the request context, then returns required column-projection actions and row-filter expressions that the reading engine must apply as it reads [6]. What the engine gets back is one principal's outcome, expressed as filtering or masking instructions, not the administrator's policy [7]. Databricks writes that delegated enforcement needs a common contract between catalog and engine, and that the Iceberg community adopted read restrictions to provide it [16].

So policy evaluation sits in the catalog and enforcement sits in the engine [5].

The contract is bounded. The initial spec defines nine predefined column-projection actions and standardized row-filter expressions such as comparisons or set membership [8]. Databricks is direct about what falls outside: many real enterprise policies depend on subqueries, lookup tables or custom UDFs, which cannot be expressed in that vocabulary, and a policy is representable only when the catalog can reduce its result to those terms, otherwise policy semantics are lost [9]. Before adopting this, a platform team has to check each existing masking rule against that list.

The other design decision matters more in a shop running several engines. The spec defines what a trusted engine must enforce, but not how the catalog establishes that trust, and a claim from the client is not sufficient, so administrators and implementations have to supply security mechanisms appropriate to their environment [12]. In practice, then, the deployment's security configuration is what decides which engines count as trusted. Iceberg community discussions have considered mTLS and OAuth, and Databricks says trust ultimately remains outside the protocol [13].

Trusted and untrusted attach to the deployment, not to the engine software. Spark and DuckDB are untrusted when users control the runtime, because those users can execute arbitrary code or access the underlying data directly [10]. A securely configured Trino deployment is Databricks' example of a trusted engine, because it provides native enforcement for row filters and column masks [11]. The same engine can land on either side of the line depending on who can reach the process [17].

The alternative keeps every step inside the catalog's environment. Databricks implements fine-grained access control on dedicated compute by transparently routing queries through a secure filtering fleet [3]. Unity Catalog's Cross-engine ABAC extends that to other engines by putting the filtering fleet behind the Iceberg REST scan and plan APIs, sanitizing data before an external engine such as Spark or DuckDB processes the result [4].

Databricks lists the implementation questions still unresolved: how an engine securely propagates the end user's identity and attributes, how a catalog distinguishes the user from the engine acting on the user's behalf, and how credentials are bound to their intended recipient [15]. On catalog labels, the second addition, the available description is a single line, that they make governance context portable across catalogs [2]. Read restrictions, the post says, are best suited to direct engine-to-catalog access where the source catalog's policies are simple and a trusted engine can enforce the resulting decision [14].

What to watch

  • Whether the Iceberg spec absorbs a trust mechanism such as mTLS or OAuth instead of leaving it to each deployment.
  • Which engines ship read-restriction enforcement, and whether any engine other than Trino is presented as trusted.
  • A published description of catalog labels detailed enough to test the cross-catalog portability claim.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories