LiveNerf measures Claude Opus 5.5 drift every day on 78 questions the model sometimes misses

The calibration step is the part of the design I would copy. Opus 5.5 answered about 93% of the 2,336 candidates correctly on the first attempt, and 97% of them came back the same way on all four samples ^[15]. A question answered right four times out of four says little about a small change in the model. The 78 left in the middle are 3.3% of the pool ^[1]. They are the questions Opus 5.5 sometimes gets right and sometimes misses ^[16], and a small loss of reasoning has room to show there.

A reasoning model cannot be pinned the usual way. There are no sampling parameters to fix and no switch for the internal thinking ^[9]. LiveNerf freezes what surrounds the model instead: the prompts, the Claude Code CLI at version 2.1.280, and the harness hash ^[10]. The project says that with those fixed, any remaining difference has one explanation: the model changed ^[12].

The claim holds for the client side. From the user's seat, the causes the write-up lists (quantization, routing to a cheaper variant, lower reasoning effort) all count as the model changing ^[4]. Sampling noise also survives the freeze, and separating it from real change is the job of the statistics. Each question is compared against its own baseline, with errors clustered by question, so item difficulty cancels instead of mixing into the signal ^[13].

The effort sweep needs a careful reading. According to the project's published validation, low effort cut output tokens 62% while the score fell 8.3 +/- 4.5 points, and medium effort cut tokens 26% while the score fell 4.2 +/- 3.9 points ^[6]. Read as 95% intervals, both drops exclude zero, the medium one by 0.3 points ^[2]. Read as one standard error each, neither drop is two standard errors from zero: the low-effort drop is 1.8 and the medium one 1.1 ^[5][6]. If each question is scored once per run, one question is worth about 1.28 points ^[3], and the medium-effort drop is roughly three questions ^[4]. The write-up's case for tokens as the early warning rests on this gap: compute spend moves before accuracy becomes statistically significant ^[7].

When a model feels worse weeks after launch, users usually have screenshots and complaints and no baseline, and a screenshot comes without an error bar ^[3]. The 'nerf' label comes from the accusation that Anthropic degrades its models days or weeks after release; the write-up says it could be true, statistical noise, or a mix of both ^[5]. It does not report whether Opus 5.5 has moved against its launch baseline in the runs logged so far ^[11].