Text-to-SQL demos test the one part of the job that stopped being hard

An accuracy figure is a claim about someone else's schema. The dev.to post says text-to-SQL accuracy is respectable on small, clean, well-documented schemas and drops sharply on real ones: hundreds of tables, cryptic column names, dirty values, ambiguous join paths ^[6]. It points at BIRD, which the post says exists because results on tidy academic schemas did not hold up on production databases ^[7]. For a vendor's number to say anything about your warehouse, the question set behind it has to have resembled your warehouse on those four counts. So the post asks for the figure, the size and source of the set it was measured on, and an offer to re-run it against your schema during a pilot ^[8]. Against the line "It uses GPT-class models, so it's very accurate", its answer is that "Model quality is an input, not a measurement" ^[9].

The in-meeting version is cheap. Bring ten questions whose answers you already know, ask all ten in one session, count ^[10]. At the one second or so the post allows for a demo chart to appear ^[2], ten questions is roughly ten seconds of generation ^[25]. The scarce input is the ten answers you already trust.

Then there is the double count. The stack does not catch it. One order joins to three line items, the sum runs across three rows instead of one, and the number comes back too large with no error to catch ^[12]. The query parsed and it ran, which the post says is all most verification badges in the category assert ^[11]. The test it offers is narrow and good: ask something whose correct answer requires a distinct count, then watch whether the result fans out and whether anything flags it ^[14]. The answer to want is a product that distinguishes "this ran" from "this matches a known-correct result", and lets a human mark a query reviewed and call it back by name ^[13].

The third question is a definitions problem. "Revenue" is not a column: gross or net of refunds, booked or shipped, tax in or out, recognised when, and the model picks one reading per query based on phrasing ^[15]. Two people then carry two numbers into the same meeting, both produced by the tool, and the post is explicit that no prompt repairs this because the ambiguity lives in the business ^[16]. The remedy it offers is a semantic layer, with metrics defined once and their join path, filters and grain fixed, so the model selects a defined metric ^[17]. It treats "The model figures it out from context" as a warning sign, along with any suggestion that users write more specific questions, since that hands the definitional work to every user on every query ^[18].

Question four, on whether row-level access is enforced in the database or in the vendor's application, is where the published text breaks off, so the remaining eight are not on the record; the post calls that fourth question the one that most often changes a decision ^[20]. The post also draws the distinction that matters there: most tools have permissions, and fewer have permissions the model cannot route around ^[21]. The opening frame names a third failure class, what happens when a generated query meets a production database at month-end ^[23].

One more thing about provenance. The post tells readers to ask all twelve of every vendor, "including us" ^[22], and describes the losing outcome as a tool that "quietly produces wrong numbers in a nice font" ^[24].