Skip to content

Build1 publisher2 min readPublished

A 328-case benchmark traces injected instructions across seven kinds of agent-harness state

HarnessSafe, from Beijing University of Posts and Telecommunications with China Telecom and the Beijing Academy of Artificial Intelligence, reports that containment depends on which carrier holds the attacker's text and on which model is driving the harness.

The Engineer · Build desk

Illustration accompanying A 328-case benchmark traces injected instructions across seven kinds of agent-harness state

What happened

  • HarnessSafe is a benchmark of 328 executable cases spread across seven persistent-carrier families, run on what its authors describe as most mainstream agent harnesses.
  • Each case is written as a Persistent-Risk Lifecycle: attacker entry, retention across carriers and system boundaries, a later benign trigger, then an observable violation.
  • The authors report that containment is carrier-specific and depends strongly on the harness-model configuration, with both the harness and the model backend shaping outcomes substantially.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint A clean result on memory injection no longer stands in for the rest of the harness, because carrier-specific containment means a pass on one surface predicts nothing about session summaries or shared artifacts.
  • decision Changing the model behind an unchanged harness becomes a security decision that needs its own re-test, and the same eval cannot sign it off as a quality-and-cost swap.
  • exposure Incident review has to start from stored state, since the transcript that shows the unauthorized tool call holds a benign request and no payload.
  • capability Staged traces give an engineer somewhere to put the fix, because the evidence names the write that carried the instruction.

The authors' example is a plain one. A compromised tool returns output with a hidden instruction. The harness stores that output in a project file or a memory entry. On a later task the harness loads it, follows the instruction, and calls a second tool to perform an unauthorized action [8]. By the time the action fires, the original malicious input may no longer be in the active context, and the request that triggered it is benign [9].

A harness, in the paper's terms, is the runtime layer that stores state, loads tools, and drives the model's execution loop [10]. Every write in that layer is a candidate carrier. HarnessSafe splits them into seven families: memory, skill, Tool/MCP, memory-to-skill transformation, subagent delegation, session summary, and shared-artifact reuse [2]. The authors group those into core carrier surfaces, cross-carrier transformations, and cross-boundary propagation [12], so the memory-to-skill family is text that entered as a note and is later loaded as a procedure.

The scoring is worth copying. The evaluation is multi-stage and trace-based: it uses observable execution evidence to say how far the chain got and where it stopped [4]. The authors' position is that attack success rates cannot reflect distinct lifecycle progression patterns [6], and their own example shows why. In a run that fails there is no payload and no malicious user, so a single pass/fail bit leaves an on-call engineer guessing which write to fix.

Three hundred and twenty-eight cases across seven families averages about 47 per family [14]. The paper's abstract and introduction report the direction of the result without per-harness or per-carrier numbers [16], so the useful part today is the finding itself: containment is carrier-specific and depends strongly on the harness-model configuration, with both the harness and the model backend substantially shaping outcomes [5].

For that to transfer to your stack, the bindings have to match. The cases keep their security semantics while binding to each harness's native storage and tooling [13]. So the question for your setup is whether your memory store, your skill loader and your artifact cache are the same surfaces the case was bound to. The model half needs its own re-test: if the backend changes containment, then a backend swap is a safety change, and the eval that cleared the old pair does not clear the new one.

Taken at face value, the result sets the reporting unit. One containment number per carrier family per configuration is seven numbers for each harness-model pair [15]. Existing benchmarks, by the authors' account, generally cover a few carriers or a few harnesses, and end-to-end success rates reveal little about how risk propagates [7].

What to watch

  • Whether the full paper or a code release publishes containment per carrier family and per harness-model pair. A team would need those numbers to compare against its own stack.
  • Whether harness vendors start reporting injection results per carrier instead of one aggregate eval.
  • Whether promoting a memory entry into a loadable skill picks up a review step in shipping harnesses.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories