The IR sets the ceiling: why Interlace swapped a DataFrame for a sqlglot AST

Interlace's 2.0 rebuild committed to one sentence about its internals: the canonical intermediate representation is a sqlglot AST plus an Arrow schema, the canonical wire format is an Arrow `RecordBatchReader`, and materialisation happens exactly once, at the sink, as a single native SQL statement executed inside the owning engine ^[5]. According to the project's own writeup, every data tool has an IR whether it admits to one or not, and that layer quietly decides the ceiling on what the tool can do ^[1].

The evidence offered is a failure, not a benchmark. An earlier iteration of Interlace effectively used a pandas DataFrame as its IR: every model boundary ran an eager `.execute()` and fed the result back in ^[3]. That single choice cost laziness, dialect portability and any prospect of semantic change detection at once ^[4] - three symptoms with one cause ^[6]. The comparison the author draws is dbt, whose IR is templated text, which is why `ref()` is a string and why a macro can emit SQL no analyser can reason about ^[2].

The clean-up was expensive. Dropping Ibis was the single largest deletion in the rebuild ^[7]. The stated reason is not quality but redundancy: Ibis was acting as a data plane moving results between models while itself sitting on Arrow, and as an expression builder that compiles to sqlglot - both of which the project had already chosen directly ^[8]. Removing it dropped a heavyweight dependency and its governance risk for zero lost capability, with remote engines now connecting over ADBC instead of Ibis backends, a narrower and better-specified contract ^[9][10].

What the parsed tree buys is concrete ^[11]. Strategies become AST builders rather than string templates: each takes a model's query expression and its target and emits a short list of SQL statements applied in one transaction, with dialect appearing only at `transpile()` ^[12]. In 0.x each strategy hand-wrote DuckDB SQL ^[13]. Because no strategy needs the model's column list, a schema change needs no hand-written migration; a definition change mints a new snapshot table ^[14]. Dependencies are parsed rather than declared: `FROM` and `JOIN` are structure, a reference matching a model name becomes a DAG edge, and CTEs are excluded because the parser knows what a CTE is ^[15]. There is no `ref()` to write - `raw_orders` is a real table reference rewritten to the right snapshot at build time ^[16].

The payoff the author says justifies the whole decision is diffing trees instead of strings: `interlace plan` can distinguish a change that alters output from one that does not, then narrow per column to the downstream models genuinely affected ^[17]. The analyser is deliberately conservative. Adding `avg(amount) AS avg_amount` is additive; anything touching existing expressions, plus `SELECT *` rewrites, `DISTINCT` and positional `GROUP BY`, is treated as breaking, and ambiguity errs toward rebuilding rather than a false skip ^[18]. On the data side, Python models receive and return Arrow, and because it is a reader rather than a table they can stream with bounded memory regardless of upstream size ^[19] - the property the old `.execute()` to pandas to `memtable()` round-trip destroyed at every hop ^[20]. pandas and Polars survive as optional extras at the edges, not as the interchange format ^[21].

This is a single project's account of its own architecture, with no benchmarks, adoption figures or outside evaluation ^[22]. Watch three things: whether the conservative classifier generates enough false-breaking rebuilds to erode the value of per-column analysis ^[18], whether ADBC really covers the engine surface Ibis backends did ^[9], and whether transpile-at-the-end holds up once a second and third dialect are in production ^[12].