Topic
Methods and tools for measuring AI agent performance, including benchmarks, test fixtures, and metrics for task success, efficiency, and reliability.
Cua's pitch is an OS-level driver that delivers clicks and keystrokes in the background on macOS, Windows and Linux. The post hedges it with "where supported by the platform". A team has to test that clause before designing around it.
Reality
Researchers held a coding harness's execution loop fixed and varied planning, tools and context management across four models, and found that what each component is worth depends on how strong the model already is.
Reality
The DeepDeck benchmark runs one task twice, once with the WebMCP bundle installed and once with the tool list forced empty, freezing the model configuration in between and leaving correctness unscored unless you supply an answer you checked yourself.
Reality
Arize and Fireworks priced ten models on what a completed command-line task costs, across 2,400 runs. The winner on that metric is an open model with the worst pass rate in the study and the thinnest coverage.
Reality
CRMArena-Pro reports about 58% single-turn success and 35% multi-turn over nineteen expert-validated tasks, and the per-skill breakdown inside those averages is the part that should decide where an agent gets pointed.
Reality
JetBrains traced a model through fifteen C# refactoring tasks and found it simulating structure with sed, git and the compiler. Wiring in Rider's real engine cut median time by 83%.
Reality
Arize and Fireworks ran ten models against 40 agent tasks and found the cheapest model per finished job also had the worst pass rate. Coverage, not price, is the binding constraint.
Reality
A controlled evaluation across four benchmarks found centralized coordination lifted financial reasoning 80.9%, while every multi-agent variant tested made strict sequential planning worse.
Reality
An arXiv evaluation across OpenAI, Anthropic and Google reports that naive full-context caching can raise latency, while excluding dynamic tool results gives more consistent gains.
Reality
Sol-Luna's supervisor could have split independent modules across parallel workers. Given a free choice across six benchmark runs, it kept the work for itself every time.
Reality