benchmark
AgentDojo is a benchmark suite testing how well LLM agents complete realistic tool-use tasks while resisting prompt-injection attacks.
No evidence-backed relationships are recorded.
Penn State researchers moved agent attack-success rates by up to 13.21 points by renaming tools, with the attack, task and policy held fixed. The size of the shift varied by model and benchmark, so a robustness score only describes the tool names and schemas it was measured on.
Reality
Meta's Prompt Guard 2 caught 6 of 629 buried attacks at a 0.5 cutoff and 621 at 0.003, in a benchmark posted on dev.to. Health checks pass the same at both settings, so only attacks sent through the deployed threshold show which one a team runs.
Reality
One developer scored ten open-source prompt-injection detectors against 629 AgentDojo attacks and found the best caught about half. His own gate at the tool boundary then held the legitimate payment as well as the injected one.
Reality
The AgentDojo framework pairs 97 tool-using tasks with 629 security test cases in one stateful environment, and current LLMs solve fewer than 66 percent of those tasks with no attacker in play. That baseline bounds what its defense numbers can mean.
Reality
The paper's enforcement point reads a whole session's provenance before committing any action, so a read-email, call-payroll, send-externally sequence gets judged as a flow. Adoption means instrumenting every execution path.
Reality
The Agentic AI Detection and Response system runs in production at Uber. Three of its five components are now open source, including a 300-task, 133-MCP-server benchmark.
Reality
A framework running at Uber for over ten months argues endpoint tooling is structurally blind to agent reasoning. On the authors' own benchmark it still misses a third of attacks.
Reality