AgentMail's agent tests trace most failures to tools that leave the agent guessing

In the most common failure, the agent listed the inbox, saw nothing and concluded the email had never arrived ^[18]. Claude Opus 5.5, running in Claude Code, tried eight different ways of looking at the inbox, spam, trash and junk among them, and never asked for unauthenticated mail, the one view the documented path uses ^[8]. The default list leaves out spam-flagged mail and mail from senders that could not be authenticated, and both are one parameter away ^[6]. The report says the empty answer looks complete and nothing in it says otherwise ^[17]. Its advice to agents is "Treat an empty result as a question, not an answer." ^[14]

Here's what teams tell themselves about a failed run: the model wasn't good enough. Here's what the report shows agents doing: taking a tool's answer as complete and acting on it ^[1]. The other failure points follow the same pattern. In one, the agent trusted an endpoint's name when the endpoint did less than the name said ^[5].

The report does not clear the models. Claude models replied to spam-flagged mail 65 points less often than the 88% comparison ^[21]. The same report lists a spam label that contradicted the user among its failure points, where the agent refused a job it had been asked to do ^[15]. In our view the label is the tool's to clarify and the caution is the model's, and the first has to be fixed before the second can be judged.

Harness choice adds its own spread. Sonnet 5's two harness scores sit 31 points apart ^[22], and the report says that for some models the harness matters as much as the model ^[16]. Linking the docs helped by 8 points ^[20], mostly by helping agents find the tool ^[19].

AgentMail builds the API, wrote the tasks and graded the recordings, and its partner Scope ran every agent ^[4]. Each task rests on a documented feature that has a familiar alternative that looks like it should work ^[23]. The traps were set on purpose, so we'd expect the share of failures at each one to reflect the test design as much as agent behavior in general. AgentMail says its API stands in for agentic tools and that every trap has a counterpart in other APIs ^[24].

For a rollout this week, a failed run can be sorted on two questions. Did the tool's answer say what it left out, or did it only look complete? And did models or harnesses split on the same task? A silent tool that every model fails on needs a response that says what it omitted and an endpoint named for what it does ^[13]. A silent tool where models split needs that fix first and a model choice second. A clear tool where models split is a model or harness decision. A clear tool that every model fails on points at the work itself, which the report found was almost never what went wrong ^[2].