Published · 6d agoLeadership2 min read
A 2.6x compromise-rate spread comes from a runtime choice security teams never see
Holding the model constant, a CSO Online test measured agent compromise rates from 11.9% to 31.1% across four orchestration frameworks. The variable was the runtime, not the model.
Context for builders, not their beat.See today for builders
What happened
- Across thousands of adversarial test runs with the model held constant, the compromise rate ranged from 11.9% on the most resistant orchestration framework to 31.1% on the least, a 2.6x spread from framework choice alone.
- The test held the model constant and varied only which orchestration framework was wrapping it: CrewAI, LangChain, AutoGen and SmolAgents.
- The gap between the most and least resistant framework is 19.2 percentage points of compromise rate.
- An orchestration framework is the software layer between the model and the outside world: it decides how the agent plans its steps, when it calls a tool or API, how it remembers information across a task, and how much it can act on its own before checking back in.
- A framework that requires every tool call to pass an explicit schema check before execution gives an attacker far less room to smuggle a malicious parameter through than a framework that lets the model call a tool directly off its own generated text.
Compiled by The Board RoomSomething wrong?How this is made
Why it matters
The figure is 2.6x. Across thousands of adversarial test runs with the model held constant, compromise rates ran from 11.9% on the most resistant orchestration framework to 31.1% on the least, according to a test published by CSO Online that varied only which of CrewAI, LangChain, AutoGen or SmolAgents was wrapping the model [1][2]. That is a 19.2 percentage point swing [3] produced by a decision most organisations treat as an engineering preference.
The mechanism is unglamorous. The framework, not the model, decides how the agent plans steps, when it calls a tool or API, what it remembers across a task, and how far it can act before checking back [4]. CSO Online's example is concrete: a framework that forces every tool call through an explicit schema check before execution leaves an attacker far less room than one that lets the model call a tool straight off its own generated text [5]. The payloads were the ordinary agentic ones, including tool call hijacking, cross-tool injection, memory poisoning and delegated authority abuse [6].
It is not a lone result. Lasso Security reports that swapping one piece of agent plumbing for another moved attack success rate from 1% to 24% using the identical model, prompt and tools [7], a 24-fold difference [8], with each harness left in its default configuration because out-of-the-box behaviour is what vendors actually sell [9]. Omar Santos of Cisco puts the ownership problem plainly: the harness should be treated as an attack surface "because it is where the agent gets its authority, context, and paths to act" [10]. CSO Online describes that layer as under-inventoried, under-tested and frequently under-owned inside enterprises [11].
The size of the effect is task-dependent. The ALE-Claw study on long-horizon computer-use tasks found the model mattered about three times more than the harness, 18 points of pass-rate spread against 6 [12]; Harness-Bench concluded capability should be reported at the model-harness configuration level rather than attributed to the base model alone [13]. The larger spreads show up under adversarial pressure, where the loop that governs probing and retrying is the harness's own [14].
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Across thousands of adversarial test runs with the model held constant, the compromise rate ranged from 11.9% on the most resistant orchestration framework to 31.1% on the least, a 2.6x spread from framework choice alone.
- [2]
The test held the model constant and varied only which orchestration framework was wrapping it: CrewAI, LangChain, AutoGen and SmolAgents.
ReportedView cited source - [4]
An orchestration framework is the software layer between the model and the outside world: it decides how the agent plans its steps, when it calls a tool or API, how it remembers information across a task, and how much it can act on its own before checking back in.
ReportedView cited source - [5]
A framework that requires every tool call to pass an explicit schema check before execution gives an attacker far less room to smuggle a malicious parameter through than a framework that lets the model call a tool directly off its own generated text.
ReportedView cited source - [6]
The evaluation ran the same set of adversarial payloads, including tool call hijacking, cross-tool injection, memory poisoning and delegated authority abuse.
ReportedView cited source - [7]
Researchers at Lasso Security found that swapping one piece of supposedly neutral agent plumbing for another moved a model's attack success rate from 1% to 24%, using the identical model, prompt and tools.
Sources & coverage · 2 publishers
The reporting this story was synthesized from, earliest first. Every link goes to the original.
- csoonline.com6d agoThe AI harness is the new attack surface | CSO Online
- lasso.security6d agoClaude Agent SDK vs. deepagents: Red-Team Results
Cited in this coverage: Lasso Security, reported by CSO Online
- csoonline.com6d ago
