Four readers sent an LLM triage experiment back for a control arm and frozen predictions

The objection lands differently on each of the three models. Gemma's rejections split 51% on ground-truth false alarms against 2% on real bugs ^[5], a gap of 49 points ^[4]. A judge that rejects because it cannot see bugs fails symmetrically, throwing out real vulnerabilities and false alarms at similar rates, because it cannot tell them apart ^[6]. So the published tables answer one half of the complaint; on the other half, sycophancy and over-reporting both fit the same rows for gpt-4o-mini, and the published numbers cannot separate them ^[7].

In his own terms, the commenter set it out this way. "Every prompt already tells the model that scanner flagged this, so you measure agreement with the claim and detection together, in one number," one reader wrote ^[4]. He then added a constraint on procedure. "Write the prediction down and publish it together with the result, because once numbers exist both stories explain them equally well and it becomes very easy to pick the one that fits," he wrote ^[9]. The author's response is that the sentence describes half the AI benchmarking content on the internet ^[24].

Stripping the flag took more than a one-line edit. Designing the neutral arm showed the anchor sitting in three places in the prompt ^[12]. The vulnerability class stays named in both arms, which was the author's one refinement to the reader's design: remove the class as well and two variables move, since verification becomes open-ended detection ^[13]. The flagged arm is produced by the exact code path that generated the published numbers, asserted byte-for-byte on every candidate ^[14]. Without that assertion, drift in the prompt between the two runs would arrive as an anchoring effect.

The frozen predictions are specific enough to fail. Confirms by gpt-4o-mini on the safe subset should fall by at least 15 percentage points once the flag is gone ^[15]; against a flagged-arm rate of about 80% on those slices that means 65% or lower, and on 100 ground-truth safe cases it means a net 15 slices flipping from confirm to reject ^[1][2]. Gemma should move by less than 10 points ^[16], the drops should order mini, then gpt-4o, then Gemma ^[17], and all three should hold at 95% or better on the real-bug subset ^[18]. The exit was written down too: a negligible drop for mini makes it an over-reporter, not a sycophant, and the earlier article's causal framing gets corrected in those words ^[19].

Next came the statistics. The frozen rule called any 5 to 15 point difference inconclusive, reasoning from single-rate noise of roughly plus or minus 8 points at n=100 ^[20]. A second reader pointed out that both arms judge the same 200 cases, so the design is paired and the information sits in the cases that change answer ^[21]. "What carries the signal is the count of cases that flip confirm-to-reject against the count flipping the other way," he wrote ^[22].

For the difference between the two arms to mean anything on someone else's findings, their prompt has to carry the anchor in the same three places this one does ^[12], and their labels have to be as clean as the 100-and-100 split the OWASP Benchmark supplied here ^[11]. The review also delivered work the author did not ask for: across four days, four readers pinned his model versions and wrote a production monitoring spec ^[23]. The results of the flag-free run are not in the portion of the writeup available for review, though the author says it publishes the predictions that survived and the ones that did not ^[25].