A fifteen-line abstention rule removed more correct answers than confident wrong ones

Grading is mechanical. A confabulation here is a non-abstained answer the grader marked wrong, and the post claims no intent ^[4]. The layer itself is fifteen lines, deliberately thin, cut from a larger verification system the authors say they run in production on themselves ^[2].

Across the four arms that ran it, confident wrong answers fell 18.4 points while correct answers fell 22.6 points ^[1][2]. That is 1.23 correct answers surrendered for every confident wrong answer removed ^[3]. If right and wrong are the grader's only two non-abstention verdicts, the share of items those four models were willing to answer at all went from 69.7% to 28.7% ^[4].

The correctness loss comes from retrieval. The layer retrieved from Wikipedia, which often did not contain the answer, and the rule then correctly drove abstention ^[6]. The authors state it plainly: the wrapper's accuracy is bounded by retrieval quality ^[8]. This copyable layer cannot go looking for more evidence, which the post says is deliberate, and the larger system it was cut from does that half of the job ^[7]. For the honesty delta to arrive without the correctness loss, your retrieval has to cover your questions better than Wikipedia covered theirs.

DeepSeek resisted. Wrapped, it moved from 33% to 24%, nine points, while three of the others collapsed to near-zero ^[9][5][20]. On Vectara's summarization-faithfulness leaderboard, the post notes, those same models order the other way, with DeepSeek mid-pack and ahead of Grok and Gemini ^[11]. A contemporaneous paper, LegalCiteBench, found that abstention-prompting alone did not fix legal-citation correctness ^[12].

Anyone shipping a wrapper has a harder case with Claude Fable 5. Every one of the 295 wrapped prompts returned stop_reason: refusal while every bare prompt answered normally ^[13], and a fresh two-item retest days later reproduced it ^[14]. A reworded prose version of the same instructions answered fine, which the authors read as evidence that the prompt's structured shape is what triggers the refusal; one intermediate probe was recorded answering on one occasion, so they stop short of calling the trigger diagnosed ^[15]. A team shipping this layer has to plan for a model that refuses the wrapped prompt outright.

For these numbers to transfer, a team's grader would have to mark answers the way this one does, and its question mix would have to look like a corpus assembled from documented failures. The post is explicit that the bare rates are high by construction, that they say nothing about these models on ordinary traffic, and that the finding is the bare-to-wrapped delta on identical items ^[17]. The grader is the authors' own rather than any lab's official scorer, and every number comes from one sampled response per item ^[16]. That grounding plus abstention reduces confabulation was already established by Stanford's Large Legal Fictions in 2024 and the RAG-faithfulness literature ^[18].