benchmark
BIRD is a large-scale cross-domain text-to-SQL benchmark with real-world databases, used to evaluate natural-language-to-SQL systems via execution accuracy.
No evidence-backed relationships are recorded.
Frontier models average 59.5 on Argo-Bench and clear 95 on only 34.8% of its 210 enterprise data tasks. The benchmark grades the bans and refunds an agent files against a hidden simulator, a step outside what text-to-SQL scores measure.
Reality
A buyer's checklist published on dev.to puts accuracy measurement and permission enforcement ahead of the live demo, and every question on it comes with a test you can run inside the meeting. Its author sells in the category.
Reality
Yang Fei and three co-authors add roles and column-level policies to Spider, BIRD and LiveSQLBench, then score existing systems on a failure class that covers queries returning the right rows to a user barred from the column.
Reality
Thinking Machines and four academics trained one model past the usual agent pipelines on text-to-SQL, but the part worth copying is the audit they ran first, which put annotation errors in 61.1% of the benchmark examples they checked.
Reality
A paper on arxiv.org names the failure the Echo Gap: wrong episodes get inflated scores, then get retrieved more. The proposed fix needs a verifier whose errors are uncorrelated with the first grader's.
Reality