Science1 publisher3 min readPublished
The UK's AI Security Institute read 6,390 transcripts to find out why its agents failed
The institute says an average pass rate scores a refusal and an inability the same way, so it read almost 6,400 evaluation transcripts to see which was happening. Pass rates stay in its pre-deployment reports.
The Scientist · Science desk
What happened
- The AI Security Institute published a post arguing that average pass rates should be read alongside transcript analysis, with takeaways from its own reading of almost 6,400 evaluation transcripts.
- Its case study put agents built on nine models against a private suite of 71 capture-the-flag cybersecurity tasks graded from trivial to expert.
- The post lists four limits of pass rates, among them that bugs during an evaluation can make an agent look less capable than it is outside the harness.
Compiled by The ScientistSomething wrong?How this is made
Why it matters
- constraint Ordering agents by pass rate puts one that declined the job level with one that tried and could not do it, so the ordering ranks outcomes and not safety, even when the numbers are close.
- decision A team deciding whether to spend on prompting or tool scaffolding needs the return per unit of engineering effort, and the pass rate AISI publishes measures something else.
- cost Reading transcripts that run the length of a novel is analyst labour, so the richer picture is paid for in evaluator time or in a classifier somebody has to build and validate.
- precedent A state evaluator counting refusals and policy blocks as distinct outcomes makes it harder for a developer's own report to present a single average as the whole result.
A pass rate is one number: the fraction of attempts an agent solved across a set of tasks [1]. A transcript is everything that number drops. AISI lists what one contains: the initial task instruction, the agent's own commentary messages, the tool calls it made, and the outputs those tools returned [2].
The sampling can be recovered from the case study's design. Nine models against 71 capture-the-flag tasks is 639 model-task pairs, and 6,390 samples divides into 10 attempts per pair [18]. On that reading, the agent that set off its provider's policy violation detector in 30% of attempts did so in roughly 213 of about 710 attempts [19]. Those 213 attempts record the provider's detector firing, not whether the agent could have found the flag.
The institute is keeping the metric. Average pass rates remain one of the key statistics in its pre-deployment testing, and it uses them to track capability trajectories in chemistry, biology and cybersecurity [9]. The narrower claim is the one in the post: pass rates are useful but "can flatten important nuances if used in isolation" [8], and transcript analysis supplements them [14].
Two of the four limitations the post lists are about attribution. An agent can fail because it refused in line with its safety training, or because it could not work the external tools it was given, and "These failures may not indicate capability limitations" [10]. A pass rate in isolation, AISI wrote, "says nothing about whether a model can be engineered to solve more tasks, nor what the performance returns are for each extra unit of engineering effort" [12]. Bugs cut the same way: when a harness fault blocks an agent, scoring on outcomes alone understates what the model can do elsewhere [13].
The limitation that bites hardest for anyone using these numbers as a ranking is the third. Agents with similar average pass rates can have different safety properties, with some more prone to take disruptive actions, misreport progress, or omit important information [11]. Those differences live in the transcript. AISI studied message composition, message length, and the types of commentary an agent produced [16].
The post supplies a set of named failure modes with two of them counted; the full breakdown sits in a separate case study it links to [15]. The task suite is private [4], so the rates rest on AISI's own runs. And the richer picture costs evaluator time: many of these samples produced transcripts the length of an entire novel [5].
The post names one more mode but has yet to put a number on it. Agents differ in perseverance, and some conclude early that a hard task is beyond them [17].
What to watch
- Whether the linked case study publishes the 6,390 samples broken down by failure cause and by model.
- Whether AISI's next pre-deployment report puts refusal and policy-block rates next to the pass rate.
- Whether any developer publishes refusal-adjusted scores, so the same agent can be compared across different providers' policy filters.