A verifiable reward for text-to-SQL is a comparison, not a judgement. Run the model's query, run the reference query, score on whether the results agree [16]. That makes the reference annotation the grader. In the roughly 2,500 examples sampled from BIRD Train, a widely used text-to-SQL dataset, 52.1% of reference queries were wrong [5][6]. The researchers name both failure modes: a sound query is punished when the reference is wrong, and a flawed query gets credit for matching one [20]. So the cleanup sits upstream of everything else, because otherwise the reward channel is noisy in a direction that correlates with the annotator's mistakes.
The error categories overlap, which changes how you would triage your own data. 61.1% of the audited examples carried at least one identified problem and 52.1% had a bad reference query [9][6], so 9.0 points of the sample is flagged on the strength of something else [1]: 26.2% had a flawed natural-language question and 18.2% supplied incorrect external knowledge [7][8]. The reference query is both the dominant defect and the one wired into the reward.
The first review pass tried to automate this. An OpenAI o3 model used as a screen was right 90.6% of the time when it flagged an annotation error, and it surfaced 24.5% of the errors human reviewers found [10][11]. Invert the recall and the humans turned up about 4.1 times as many defects as the screen [2]. Precision that high makes the screen useful for ordering a review queue; recall that low means it cannot close the queue, which is why the corrected set, BIRD-Platinum, came out of multiple rounds of expert review with conflict resolution [12].
On the model side the two headline numbers are 88.55% with greedy decoding and 91.37% when selecting among 16 candidates [3], which is 2.82 points for sixteen times the sampling [3]. Whether that trade is worth making depends on whether you are filling in a leaderboard row or a latency budget.
For any of it to transfer, the reported setting has to resemble yours. This is a demonstration on curated academic test sets, and the paper reports no customer deployment and no evaluation against a live corporate database [18]. Production warehouses add access controls, schemas that move, organization-specific terminology and column counts far past the benchmark's [19]. The training route itself is unremarkable in the good sense: reinforcement learning on Moonshot AI's Kimi-K2.6 through Tinker, which handles the compute and exposes sampling, gradient calculation, weight updates and checkpointing [14][15]. The hard-won artifact is the labelled data underneath it. If your own eval labels have not been through comparable review, the accuracy you report includes your annotators' accuracy as well as your model's.
