Start with the interval, since it does the real work in this pilot. Newcombe put the treatment-minus-baseline difference at 0.000000, with a 95% interval of [-0.203883, 0.203883] [3]. That window is about 41 percentage points wide [1]. Scale the half-width by the 15 attempts in an arm and you get 3.06 attempts [2]. So the design cannot exclude an effect worth three passes out of fifteen, in either direction. Two identical perfect scores are easy to mistake for a finding when they are not one.
Part of that comes from the scoring rule. An attempt passes only as the logical AND of eight checks: artifact contract, source integrity, visible and hidden sensitive-value removal, secret and path removal, preservation of safe content, byte-exact Markdown, and byte-exact findings JSON [6]. As a release gate, that is the right shape. As a measuring instrument, it discards everything except the final bit. Once the baseline clears all eight on every attempt [1], there is no margin left to record, and the finest difference an arm can express is one attempt, or 6.7 points [3].
Two conditions would have to hold before that 15/15 said anything about your agents. Your task family has to fail at the baseline often enough for 15 attempts to see it. And your agents have to be cued: both arms got the same generic instruction to identify and read applicable skills, so what was tested was encouraged availability rather than spontaneous discovery [8]. Two further limits are the author's own. The inputs were reserved synthetic examples, and he says the result should not be read as evidence of real-world PII safety [9]. The intervals describe repeated runs on three fixed fixtures and do not estimate generalization across the broader population of public-release tasks [16].
The cheap fix is ordering:
1. Run the baseline arm alone against candidate fixtures. 2. If it saturates, harden or replace fixtures until it fails at a rate 15 attempts can resolve. 3. Then freeze the protocol and spend the treatment arm.
The harness deserved a better benchmark than the one it got. Protocol, runner, skill, fixtures, order and analysis rule were all frozen before the scored batch, and a context-isolated adversarial review had to return GO before the one-shot runner could start [7]. All 30 trials came back healthy, with no diagnostic failures, infrastructure errors, retries or replacement attempts [10]. Most usefully, the run splits one question into three: whether the procedure was available, whether it was used in the intended order, and whether using it actually improved the outcome [17]. The first two were answered.
What stays unmeasured is the part the reviewers flagged: creating, testing, loading, selecting, updating and retiring the skill [13]. The pilot prices none of it, and the author declines any break-even claim because human design and maintenance time was never measured [15]. My reading, which I would revise given a discriminating fixture set: an internal agent eval needs a published baseline failure rate before it can claim to measure anything. This one at least says so, in the author's terms, that the overhead of the procedure is observable while its outcome value remained unmeasured [19].
