Topic
Methods and benchmarks for testing LLM-based agents on multi-step, tool-using tasks, covering completion, tool recall, and retrieval accuracy metrics.
One team running a GPT-5.4 agent in n8n traced its production 'laziness' to three workflow bugs, the first a retry cap cut from 6 to 2. Fixing the loop restored quality on the same model, so traces and stop reasons should be checked before any model swap.
Reality
Frontier models score at most 0.17 on a synthetic support-agent test of trusting only officially labeled claims; a two-line rule scores 1.00. A careful model that acts on rumors once their label is stripped makes the case for enforcing the check in harness code.
Reality
Both field test reports pointed at replay and matcher calibration, but v0.3.0 fixed the recall denominator with one list comprehension that scopes each candidate's references to its own domain, and the reported number roughly doubled.
Reality
Three of the four layers in a dev.to testing pyramid for MCP servers are plain pytest checks on schemas, error envelopes and session expiry. Only the fourth puts a model in the loop, and the available text breaks off before describing it.
Reality
Deferred tool loading assumes retrieval puts the right tool in the shortlist. A BM25 harness over 100 synthetic enterprise tools and 200 tasks says it does that 5 percent of the time when the user does not use your words.
Reality
Bo Qu and Mingguang Chen seal every gather-analyze-decide-execute-reflect cycle into a recompute-verifiable audit chain and score five capability axes from it. In four weeks of live paper trading the gap came to 0.23.
Reality
CauterRule's own field test puts the undecided bucket above pass and fail combined, and the report names the matcher that produced it as its top calibration target. The per-model rates end up measuring trigger phrasing.
Reality
A cash-and-stock deal moves a category-leading point tool inside a platform vendor's roadmap. Buyers mid-procurement should reopen the pricing conversation before renewal.
Reality
IBM Research ran self-mined guidelines across eight models on AppWorld. One model gained 16.1 points for 5 percent more tokens; another gained nothing at all.
Reality