The zero-gradient part deserves a walkthrough, because it explains how the dashboards were honest and useless at once. GRPO samples several answers to the same question, scores each one, and pushes the policy toward the answers that beat their group's average [16]. In this run the rewards were a formatting check worth 1.0 that fired for essentially every answer and an exact string match worth 2.0 that fired for almost none, so a group of four usually scored [1.0, 1.0, 1.0, 1.0] [19]. Every answer is exactly average, so the advantage is zero and the update is zero [20]. The reward curve reads 1.0 because formatting is the only thing left that it can measure. The metric tracking that condition hit 1.00 by step 160 of a 300-step run [21], which leaves 140 steps, 47% of the schedule, spending gradient on nothing [1]. A dollar an hour buys a lot of that [2].
Exact string matching also fails on its own terms: it scores `age>56` as zero against `age > 56` [22]. Meanwhile the training data handed the model a schema already trimmed to the columns the answer needed, which removes the table-selection work that is the actual task [17], and the run was bounded by step count rather than passes over the data, so it saw about a tenth of it [18].
The harness that fixed this runs both the candidate and the reference query against the real database and compares the rows that come back [8]. That makes correctness a fact rather than a judgement [9]. It is also, unchanged, the reward function. One artifact, two consumers: the held-out score and the RL signal. That is why the author calls the missing held-out evaluation the real defect from which the other two follow [24]. Bad data and a whitespace-sensitive reward are both invisible until something can execute SQL in a loop.
The arithmetic on where the points came from is the useful part. At 0.5B, real schemas took it from 6.4% to 44.6% [10], and a proper execution-based reward took it to 49.7% [11]. That is 43.3 points at a fixed parameter count [2]. Swapping in Qwen2.5-Coder-1.5B added 18.4 [3], so the data-and-metric work returned about 2.4 times what tripling the model returned [4].
Then the voting. Eight samples per question, majority vote over the returned rows, 71.5% [13], against 71.2% for Qwen2.5-Coder-7B-Instruct [14]. Eight passes through 1.5B parameters is 12B parameter-passes against the 7B's 7B, roughly 1.7 times the decode work [5], plus eight executions per question, because the vote is over rows and not over query strings [13]. Spider hands you a database to execute against [1]. In production that read path is yours to provision and the latency is the user's.
For 71.5% to transfer, four things have to be true: your schemas resemble Spider's 200 SQLite databases, your questions resemble its 10,000 human-written ones [1], row match is your definition of correct [9], and a 0.3-point margin from a single unreplicated run is a margin you would plan around. The author reports the 1.5B losing to the 7B by 3.1 points at one attempt [15], which is more than ten times the margin voting buys [6]. The larger gap is the more load-bearing number.
