Skip to content

Science1 publisher3 min readPublished Updated

A one-line summarize request slips code execution past Claude Code's Auto Mode classifier

Embrace The Red reports 60 to 80 percent success on a small sample where an Anthropic-commissioned evaluation scored 0.00 percent across 720 runs, and the difference is mostly in what each test could measure.

The Scientist · Science desk

Photograph accompanying A one-line summarize request slips code execution past Claude Code's Auto Mode classifier
Photo: embracethered.com

What happened

  • Auto Mode, which swaps Claude Code's human approval prompts for a safety classifier, has been the default starting mode since mid-August.
  • An Embrace The Red post reports that a plain request to summarize a website hijacked Claude Code Opus 5 in Auto Mode and reached code execution at a 60 to 80 percent success rate on a small sample.
  • Anthropic commissioned Trajectory Labs to run 72 indirect prompt injection scenarios ten times each, and the chart shared from that work shows 0.00 percent attack success for Opus 5 in Auto Mode.
  • Anthropic's Boris Cherny posted that three layers, model training plus input probes plus an intent classifier, could reduce indirect prompt injection on unseen attacks to approximately zero.
  • The server answered Claude's WebFetch call with a 415 Unsupported Media Type, after which the model decided on its own to retry the fetch with curl through the Bash tool.

Compiled by The ScientistSomething wrong?How this is made

Why it matters

  • constraint A zero across 720 pre-specified runs bounds that suite and nothing else, so anyone reading the chart as evidence that an adaptive attacker cannot get through is pricing a quantity the evaluation never measured.
  • decision Teams running the default mode are now making the isolation decision by omission: with approval prompts replaced by a classifier, no human is in the loop when the agent switches from a fetch tool to a shell.
  • exposure Any workflow whose ordinary job is to summarize a supplied URL hands an attacker-controlled server a chance to steer tool choice, which puts routine research and triage agents in scope rather than exotic setups.
  • contradiction Both numbers in this story are self-reported and neither has an auditable denominator, which means the honest read is that the true rate is somewhere above zero and no one has yet published a figure you could bet on.

The chain's most instructive moment is the one where the model did the right thing. The README inside the ZIP advertised a bundled binary, `decoder-darwin`, as the catalogue decoder [10]. Claude declined to run it and wrote its own Python decoder instead [11]. It then ran that decoder from inside the directory it had just unpacked, where a file named `struct.py` shadowed Python's standard implementation, so the decoder's import of `base64` executed the attacker's file [12]. Claude refused the untrusted binary but ran its own replacement inside that same untrusted directory, and that is where the safety broke down.

The sequence works without ever instructing the model directly. The site presents itself as a small archive of notebook records, with catalogue metadata, checksums and seven short records about the development of the theory of language, which gives the agent a legitimate reason to keep digging [17]. A 303 redirect from the root URL points `curl` at the archive [9], and Claude extracts it into a temporary scratchpad folder [15]. The post's own framing is that the attack makes the malicious path the one worth pursuing rather than telling the model what to do [14]. The actions an intent classifier would have to score here are a `curl` fetch and a Python run, both plausible ways to satisfy the sentence the user typed [7].

The two success rates are not measuring the same quantity. Seventy-two scenarios at ten runs each is 720 trials [6], and a 0.00 percent cell means none of them landed, which is a genuine result about a pre-specified suite of unseen attacks [4][5]. It says nothing about an attacker who watches which step fails and rewrites that step, which is what the post set out to test [18]. The researcher's own denominator is missing too: 60 to 80 percent arrives with the phrase "small sample size" and no count [2]. Neither figure is independently checkable, since the vendor evaluation has no published benchmark name [5] and the test endpoint serves the payload only to allow-listed IPs, with the domain partly redacted to keep it out of search indices [7].

The open question is how durable the chain is. Every specific step in it is patchable, and the 415 nudge is not even always needed, because Claude sometimes reaches for `curl` unprompted [16]. The general shape is harder to patch at the classifier: an agent that can shell out, write attacker-supplied files to disk, and then run an interpreter inside that directory [11][12]. That shape calls for isolation and monitoring, not a scoring step that stands in for an approval prompt [1]. Embrace The Red's stated conclusion is that Auto Mode is not a substitute for running the agent in an isolated environment and watching what it does [13], and the evidence carries it at the level of existence rather than rate. Treat 60 to 80 percent as a demonstration that the true number is above zero, not as a measurement of how far above [2][3].

What to watch

  • Whether Anthropic publishes the Trajectory Labs suite under a name, with per-scenario results and the Auto Mode run count.
  • Whether the chain still lands after the intent classifier is retuned, and whether the researcher publishes a sample size and a reproducible endpoint.
  • Whether the import-shadowing step gets closed at the tool layer, by refusing to run interpreters inside freshly fetched directories, rather than at the classifier.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories