Frontier models score at most 0.17 on a source-trust test that a two-line rule passes perfectly

The perfect 1.00 is a control, and it is the best-engineered part of the post. The benchmark defines correct behaviour as the provenance rule, so an agent that implements the rule passes by construction ^[5]. The author also ran the opposite baseline. A tone agent that trusts any official-sounding text scores 0.00 ^[6]. "A benchmark is only meaningful if a correct strategy can pass it and a wrong one fails," the author wrote ^[7].

With both baselines at the extremes, the laundered condition is the useful experiment. The claim text stays fixed and only the source label is removed ^[4]. Claude's laundered count means it acted on 58% of the rumors it had refused when they carried a forum label ^[9]. "Models decide what to trust based on how text is worded, not where it came from," the author wrote ^[16]. Gemini and GPT-nano never got far enough for laundering to matter, because they did not check the source in any condition ^[21]. On the author's discrimination score, official minus forum ^[13], their reported counts give a margin of 5 scenarios out of 24, against 17 for Claude ^[14].

Both failure modes put the trust decision inside the model, made by reading prose. The one agent that scored 1.00 made it by reading a field ^[5]. I think the check belongs in the harness for any tool that moves money. Before a credit, discount or return call executes, harness code looks up where the justifying passage came from. It refuses the call unless the source is official. The model still writes the reply. An unlabeled passage defaults to untrusted, so the laundered case fails closed. The author put the risk plainly: "tone is the one thing an attacker fully controls" ^[17].

The rule has one precondition. The label must be set by the retrieval layer from where the document was stored, and never parsed out of the document's text. If a customer can type "Source: official policy" into a ticket, the harness gate is a tone check with extra steps.

The numbers are narrow. The post, an entry in the Kaggle x DEV Benchmarks Challenge ^[1], uses 24 scenarios across four invented companies ^[2]. The author made it fully synthetic to avoid training-data contamination ^[20]. Each count moves in steps of one scenario, about 4.2 percentage points ^[19]. The published excerpt does not give model versions, prompts, or how the 0.17 headline score is calculated.

For the counts to transfer to a production agent, its retrieved context has to mix company policy with customer-written text. The author argues that support agents, RAG systems and tool-using agents all work from that mix ^[18]. Some of the context has to arrive labeled and some not. And the model has to be able to call an action tool with no human approving it.