Build2 publishers2 min readPublished Updated
AWS embeds DuckDB in Aurora PostgreSQL to query Iceberg and Parquet in S3
AWS has embedded DuckDB in Aurora PostgreSQL from versions 17.11 and 18.6, so one SQL query can join live rows with Iceberg and Parquet tables in S3. Reverse-ETL jobs that exist only for that join can be retired if the database instance has room for the lake scans.
The Engineer · Build desk
What happened
- DuckLabs, the team that maintains DuckDB, recently joined Amazon, and AWS calls the Aurora feature an example of that engine being built into its services.
- Setup means attaching an IAM role with the AuroraAnalytics feature, enabling the aurora_analytics extension, then creating foreign tables over the Iceberg or Parquet data.
- Readable sources are Iceberg tables in the AWS Glue Data Catalog plus Parquet and Iceberg data held in Amazon S3 or S3 Tables.
- A function called aurora_analytics_stat_statements() reports, for each query, the rows scanned, the bytes read from S3 and the cache hits.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability A check against archived lake history can run inside the transaction that is writing the new row, before that row commits.
- constraint A team whose Iceberg tables sit in a non-AWS REST catalog has to add a Glue federation registration before Aurora can query them.
- constraint Clusters on major versions before 17, or on 17 below 17.11 and 18 below 18.6, need an upgrade before the extension is available to them.
- precedent AWS says future DuckDB improvements can reach Aurora and other AWS services, so the same embedded engine is likely to appear in more of its products.
AWS says query processing "stays within Aurora, with no additional network hops" [6]. Frequently accessed lake data is cached in the Aurora instance, so repeat queries against it return faster [12]. One query can also read live operational data "including uncommitted writes" next to the lake tables [5]. For that to work, the embedded engine has to see the PostgreSQL session's own transaction state. I think it is the best-engineered part of the design.
The documented route to an Iceberg REST catalog runs through AWS Glue Data Catalog federation. The external catalog is registered with Glue once. Its tables then get foreign tables the same way Glue-native tables do [9]. From there, one query can join Aurora rows with Iceberg tables registered across several catalogs [10].
AWS describes the pipelines this is meant to replace as duplicating data, raising infrastructure costs and needing ongoing engineering effort to stay synchronized [14]. It also pitches the feature at AI agents, arguing that it is impractical to predict and pre-replicate every dataset an agent might need [15]. The argument works just as well for a human analyst with an unplanned question, though agents make better launch copy.
The efficiency case rests on predicate pushdown and column pruning. AWS says they limit reads to relevant data and keep queries efficient "even as the underlying data grows" [11]. Its demonstration is a psql session that runs `CREATE EXTENSION aurora_analytics;` and then builds a simple financial scenario [17]. That is a small example on AWS's side. For the efficiency claim to hold in production, the join a pipeline feeds today has to read a small slice of the lake per call, or keep landing in the cache. Where the join pulls most of its bytes from S3 on every request, I would keep the local copy. The per-query statistics function shows which of those cases applies before anything is switched off [13].
What to watch
- Whether AWS adds a direct Iceberg REST catalog connection to Aurora that skips Glue federation.
- Pricing and instance-sizing guidance for aurora_analytics lake scans once teams run them against production primaries.
- Which other AWS services AWS names next as carrying the embedded DuckDB engine.