A 12-task suite leaves each holdout task worth 20 points of pass rate

The split script in the dev.to post refuses to run on fewer than twelve task directories, exiting with "Need at least 12 tasks before a holdout is meaningful." ^[15]. The default holdout fraction is 0.4, and the holdout size is computed as max(5, round(len(tasks) * frac)) ^[16]. At the twelve-task floor that gives five holdout tasks and seven tune tasks ^[1]. One holdout task is then 20 percentage points of whatever rate you publish ^[2].

The max(5, ...) guard never fires at the default fraction, since any suite that clears the twelve-task gate already rounds to five or more ^[6]. It matters when someone lowers the fraction: twenty tasks at --holdout-frac 0.2 rounds to four, and the floor lifts that to five ^[4].

Five tasks can only produce pass rates that are multiples of 20 ^[7]. The scenario the post opens with is a 13-point move, 58 to 71, after three prompt edits against the same tasks ^[1][2][5]. A five-task holdout cannot report that move at all; the smallest change it can express is 20 points ^[7]. So a team that cares about single-digit deltas needs a suite well past the floor, and the post's rules make growth one-directional: you can add tasks later, but you may not add holdout tasks after you have watched them fail ^[8][18].

Exclusions are where the denominator moves. Timeouts, checkout errors and missing interpreters are not fails, and the post's fourth authoring rule says they leave the denominator ^[14]. The example meta.json sets timeout_sec to 120 ^[10]. An agent that runs past that budget on a holdout task therefore does not lose a point, it removes the task from the sample. Two exclusions out of five leaves three scored tasks, and each one is worth 33.3 points ^[3]. The same protocol demands a frozen denominator as one of its three prerequisites for calling a percentage a measurement ^[5]. In my view the exclusion count has to be printed beside the rate, or two numbers computed over different denominators get compared as if they were one measurement.

The fourth frozen object is the agent command line you will actually quote ^[6]. I would expect teams to skip that part, because it puts retries and stop conditions inside the number: prompts, tools, retries and stop conditions are all knobs on one policy, and turning them against the same tasks is ordinary overfitting ^[4]. Change the retry count and you have a second number on the same frozen tasks. The post's description of the alternative is one line: "The worksheet became the exam." ^[3]

Nothing in the protocol establishes that the holdout looks like the work your agent actually lands. It grades tasks in isolation, each one a directory with a prompt, a repo, hidden tests and an oracle patch ^[9], and the post argues for twenty well-formed tasks over two hundred demos ^[18]. For a holdout rate to transfer, those twenty would have to match the languages, repo shapes and time budgets of your production changes. Cheap inference and a spare server enter only at the end, as somewhere to rerun the frozen pass ^[19]. The split tool itself is presented as a proposed local script, run once with the manifest committed ^[20].