Adding both the guard and sandbox moved measured attack success from 16% to 20%

Sixteen percent of 25 attack prompts is four samples, and 20% is five ^[1]. One attack prompt therefore moves the headline rate by 4 points, and one benign prompt moves the false positive rate by 10 ^[2]. Per class the resolution is coarser: five prompts each, so the RAG poisoning slide from 80% to 100% is four of five becoming five of five ^[3][21]. Temperature is 0, the seed is fixed, and every run walks the same prompts in the same order ^[18]. That makes each verdict reproducible without increasing the number of samples behind it. The author wrote that the result is "a real measured over-hardening result, not a rounding artifact" ^[23]. The set is too small to separate that from one attack prompt changing its verdict ^[1].

The counters are the ground truth: whether an email fired to [email protected], whether the body contained the card marker ^[9]. No model sits in front of the tool calls. Two classes get a second look. When a counter misses on system_prompt_leak or rag_data_poisoning, the sample falls through to a separate model instance with its own prompt, reading the transcript ^[10]. The other three classes are scored by the counters alone ^[5]. For indirect injection, tool poisoning and the lethal trifecta, an attack that never fires send_email to a watched recipient and never carries the marker is recorded as a block ^[6].

The benign half inverts the semantics: attack_success=True on a benign prompt is a false positive, and it is reported next to attack success ^[12]. The author put the line "100% block + 20% pass = a dead product" into the report generator on purpose ^[13]. Output sandboxing is where that trade shows up: it adds a human-approval gate that can stall legitimate requests ^[22].

The pacing costs 700 seconds per configuration, 35 samples at 20 seconds each, and a baseline plus four cumulative defenses comes to roughly an hour of deliberate idling ^[4]. Most of a run is the harness waiting on purpose. That idling is the cheapest part of the design. Results persist after every sample, so a rate-limit crash resumes instead of restarting ^[15]. Each prompt gets a state snapshot and restore, so one sample's injection cannot leak into the next ^[16]. Seventy-three tests run offline with no API keys ^[25], and the same run() drives the Streamlit demo ^[26]. The post gives two attack-success figures and the per-class RAG poisoning change, and does not print the benign false positive rates ^[27].

The payloads sit outside the golden set. The runner injects the class payload from attacks/*.py into the corpus or the web results before each run, the way an attacker plants a document the agent will later retrieve ^[8]. So the columns measure four defenses against five payload families written in the same repository, on one model, gpt-oss-120b on Groq's free tier ^[14][2]. The counters key on send_email and a known recipient ^[9]. Another team would have to match that condition before 20% describes their agent: an email tool the model can address freely ^[1], and an attacker drawing from the same five families ^[2].