Shihipar's runs show Claude Code's effort level mostly buys self-testing

The extra effort went into verification. Both ends of the sanitizer comparison used the same model, Fable 5.1, with five attempts per level ^[2]. At low effort, Claude wrote the filter in roughly one pass and tested it against a single hand-written page ^[4]. At higher effort it went after its own first draft the way an attacker would. It read the parser's source code, ran a standard XSS test suite and then wrote a fuzzer on top ^[5]. One of those high-effort runs took about 33 minutes ^[5].

The failure breakdown covers Fable 5.1 attempts only ^[6], and it shows where that loop stops helping. From low to max, missed edge cases fell 59% while wrong calls fell about 20% ^[8]. Tests catch bugs in an approach that is already correct. They do not flag an approach built on a misread requirement. Wrong calls therefore grow as a share of what remains: 69% of the two main failure types at low, 82% at max ^[11].

The failure that grows is ambiguity. Wrong readings of an unclear task went from 19% of wrong calls at low effort to 44% at max ^[12]. Shihipar wrote that at higher effort Claude "will also make more assumptions on my behalf" ^[10]. The dev.to write-up ties that to runtime: the longer the agent works on its own, the more it guesses what the user meant ^[21]. It also allows a second explanation. The misreadings may always have been there and become easier to see once the sloppier failures cleared out ^[13].

All of this is someone else's workload, reported secondhand. The write-up's author says they have not measured their own setting ^[19]. Each task category holds only 4 to 20 tasks ^[18]. On a sample that size, a few flipped tasks would move the 41-point hardware gain ^[20] a long way. For the pattern to carry over to your own work, two things have to be true. First, the work has to contain hidden edge cases a test loop can reach, and Shihipar puts the hardware and security gains down to exactly that ^[15]. Second, the task has to be specified tightly enough that more autonomy does not turn into more guessing.

The category results give the per-task rule. The write-up describes operations tasks as rulebook-style work, such as filing a trade report correctly ^[16]. Those rose from 12% to 22% at top effort ^[14]. Software, which the write-up calls the closest match to everyday developer work ^[16], went from 43% to 56% ^[14]. In my view that justifies low or medium for well-specified changes and rulebook tasks. Xhigh or max belongs on code that faces adversarial input, like the sanitizer. For a vague ticket, I would spend the effort on the spec before spending it on the run.