dbt's benchmark rerun finds data modeling lifts LLM accuracy more than model choice
dbt reran its text-to-SQL benchmark, 11 questions run 20 times each, and found a well-modeled Semantic Layer gets accuracy close to 100%. For a team connecting an agent to its warehouse, that makes modeling work a better accuracy fix than a model upgrade.
Not a builder's beat, but builders have a standing stake in it.
What happened
- dbt found that adding even minimal modeling on top of raw tables improved results for both text-to-SQL and the Semantic Layer.
- On which LLM to use, dbt's own heading says the choice matters less than you'd think.
- dbt's text-to-SQL runs only worked with the entire schema loaded as context, a setup dbt says is impractical for larger datasets.
- Omni says open-weight GLM 5.3 Flash, at $0.50 per million tokens, matched Sonnet's accuracy on its internal benchmark at a twelfth of the cost.
Why it matters
- Who has to decide decision A team choosing between a pricier model and a quarter spent on modeling now has benchmark evidence from dbt that favours the modeling.
- What it costs cost Spend moves from per-token pricing to the analytics-engineering hours behind semantic definitions. A cheap model only reached parity at Omni once that structure was in place.
- What it limits constraint Semantic Layer accuracy only covers what someone has modeled. Every new metric users ask about is modeling work before the agent can answer it reliably.
- Who's exposed exposure Questions outside the modeled scope fall back to text-to-SQL. There, a query can run cleanly and still return a wrong number, and nothing between the question and the SQL catches it.
dbt reports its 2026 result in words and points readers to its repository for the detailed data and costs [1][9]. The post itself does not give accuracy percentages for each configuration. To see how far accuracy moved between raw tables and modeled ones, a reader has to go to the repo.
The test design separates context from the model. dbt ran four setups. The first was text-to-SQL on the schema. The second was a minimal Semantic Layer on the original normalized tables, and the third a properly modeled Semantic Layer. The fourth was text-to-SQL with access to the same modeled tables [14]. That fourth setup tests whether better tables help a model that still writes its own SQL.
The two approaches differ in where the model is allowed to guess. In text-to-SQL, the model works out meaning from table and column names and writes a new query every time [15]. Through the Semantic Layer, it only picks metrics and dimensions, and dbt's MetricFlow engine writes the SQL deterministically [16]. dbt wrote that this "means the LLM can't produce subtly wrong results" [17].
Omni built the same idea into its agent. Each question is routed to a curated dataset, the agent loads its semantic definitions, and the model gets tools to pick fields, filters and joins [11]. Users saw what happens when that path is shaky. During a weekend when Omni ran its own instance on open-weight models, the agent ran the same query nine times, got zero rows each time and never changed its approach [19]. Before the fixes, the Generate Query tool failed 45% of the time, and two other tools failed on every call [12]. Prompt caching later cut Omni's Sonnet bill by about 75% [13].
For Monday, sort the questions users actually ask on two axes. The first is whether the metric is already modeled. The second is whether a plausible but wrong number would feed into a decision. Modeled, high-stakes questions go through the semantic layer. Exploratory questions on small or unmodeled data can go to text-to-SQL, which is where dbt draws the line too [8]. That leaves high-stakes questions nobody has modeled. They are the backlog, and dbt's results point to modeling hours as the fix ahead of a pricier model [4][5].
What to watch
- The repo figures for text-to-SQL on modeled tables, which show how much of the gain comes from the tables themselves.
- A rerun on a dataset too large to load the whole schema as context, where text-to-SQL loses the setup it relied on here.
- Whether Omni's LLM-agnostic harness work reaches customers and open-weight accuracy holds on customer Topics.
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
dbt reran its 2023 benchmark comparing text-to-SQL and the dbt Semantic Layer with the latest generation of models, published as a 2026 benchmark update.
- [2]
The benchmark used the ACME Insurance dataset created by Juan Sequeda et al. from data.world: 11 questions, each run 20 times, across multiple LLMs.
- [3]
For queries covered by a well-modeled Semantic Layer, accuracy approaches or hits 100%.
- [4]
Data modeling quality matters enormously for both approaches; adding even minimal modeling on top of raw tables improved results across the board.
- [5]
dbt's section heading on model choice: 'Which model should you use? (It matters less than you'd think)'.
- [6]
To make text-to-SQL work, dbt loaded the entire schema as context, which it says isn't practical for larger datasets.
- [7]
The trade-off is coverage: the Semantic Layer can only answer questions that fall within the scope of what's been modeled.
- [8]
dbt recommends text-to-SQL for ad hoc analyses and smaller datasets, and the Semantic Layer for enterprise use where accuracy is critical and datasets are large, complex, or messy.
- [9]
dbt directs readers to its repo for more detailed data and the associated costs.
- [10]
Omni ran GLM 5.3 Flash, an open-weight LLM costing $0.50 per million tokens, on its internal AI benchmark; it matched Sonnet's accuracy and was 12x cheaper.
- [11]
Omni's agent does not ask the LLM to write SQL from scratch; it routes each question to a Topic (curated dataset), loads semantic definitions and context, and gives the LLM tools to select fields, filters, and joins.
- [12]
In Omni's eval, the Generate Query tool failed 45% of the time; the Generate forecast and Summarize full result tools had a 100% error rate; Visualize tool errors were 76%.
- [13]
Prompt caching cut Omni's Sonnet bill by about 75%.
- [14]
dbt tested four configurations: text-to-SQL with schema information; a minimal Semantic Layer on the original highly normalized tables; a modeled Semantic Layer following dbt best practices; and text-to-SQL with access to the same new models.
- [15]
In text-to-SQL, the LLM infers the semantics of the data from table names, column names and relationships, and writes a query from scratch every time.
- [16]
With the Semantic Layer, the LLM decomposes a question into metrics and dimensions, and dbt's MetricFlow engine handles query generation deterministically.
- [17]
The Semantic Layer's deterministic query generation means the LLM can't produce subtly wrong results.
- [18]
In text-to-SQL the LLM might join tables incorrectly, misinterpret a column's meaning, or produce a query that runs successfully but returns wrong results; there's no guardrail between the question and the generated SQL.
- [19]
When Omni switched its own instance to open-weight LLMs for a weekend in early July, a teammate posted a screenshot of the agent running the same query nine times, returning zero rows each time, and never changing its approach; Omni switched back that afternoon.
Sources & coverage · 2 publishers
The reporting this story was synthesized from, earliest first. Every link goes to the original.
- docs.getdbt.comApr 6Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update | dbt Developer Blog