Ten tasks, three runs each: grading a free coding model before it edits your repo

The failure mode worth building a harness around is the one the source ranks as both most common and most dangerous: output that looks plausible and gets rejected the moment something actually executes it ^[15]. That is why the judge in this design never reads the code. It checks behavior only, so a model can pass with code you did not expect and fail with code that reads beautifully ^[7]. The anecdote the piece opens with is the unmeasured version of the same event: a 400-line function handed over for splitting came back as 200 lines of fresh code, the build broke, and nobody established why ^[1]. An assertion script would not have prevented it. It would have converted it into a verdict before the merge.

The report the harness writes carries task name, run number, verdict and elapsed seconds ^[9]. Token usage is not in there; you are told to lift it from your client's usage field, and reminded that the free tier is counting those tokens whether you record them or not ^[10]. So the file that decides whether the model may touch code and the file that tracks what that decision cost are two different files, and only one of them appears on its own ^[19]. Three runs per task do give enough samples for the median the author prefers over the mean ^[12], which helps for wall time and does nothing for spend.

Read the results by class rather than in aggregate. Refactor failures usually point at context limits rather than raw model weakness, and multi-file tasks degrade before single-file ones ^[14]. Nondeterminism surfaces as two runs of one prompt landing on opposite verdicts ^[13]. A single score out of ten hides both, and the decision table consumes exactly that single score ^[11]. If your ten tasks are eight greenfield and two refactors, a comfortable number means less than it appears to.

Hold the free-tier figures at arm's length. They come from the vendor whose product outreach paid for the write-up, and the same write-up says the terms change and that you should confirm them in the repo before depending on them ^[2][3]. What transfers is structural. The tasks are drawn from your own workload and the pass criteria are yours to write ^[6], which means the measurement does not require trusting anyone's marketing. The thresholds are a different matter. A band that clears a model for ordinary diff review at eight of ten is somebody else's judgement about your tolerance for a broken build ^[11], and nothing in the procedure derives it from your codebase.