Graft reports 66% against 54% on SWE-bench Verified. At 50 instances a side, that is p 0.22

The arithmetic gets unforgiving at small n, and the page never says what n was ^[3]. Assume fifty instances per arm, the size at which the published rates land at p about 0.22 two-sided ^[3]. At that size, 66% is 33 tasks resolved and 54% is 27 ^[5]. The entire headline is six instances.

Pooling the arms gives a 60% resolve rate ^[2] and a standard error on the difference of just under ten points, so a 12-point gap ^[1] sits about 1.2 standard errors from zero. To push those same two rates under p<0.05 you would need roughly 128 instances in each arm ^[4]. That is grading work somebody pays for, and it is not the reader.

None of this is a complaint about the harness. Official grading settles whether a patch resolved an instance ^[1]. It has nothing to say about how many instances you need before a difference in resolve rates means something.

The odd part is the asymmetry inside one README. The efficiency claim carries its design on its face: 162 runs, same agent, same file tools, only the context differs ^[2]. The correctness claim, which the page positions as the guardrail that the agent is not getting more answers wrong ^[5], arrives with no count at all. PlanetScale's Metal page does the same in another domain, reporting an average 70% decrease in p99 and p95 latency from a few customer workloads without saying how many ^[6]. Denominators are the first thing to fall off a benchmark page.

As published, 66 against 54 supports "not worse". It does not yet support "better".