Topic
Deterministic fixtures, run modes and assertion-based harnesses used to measure agent system performance.
JetBrains traced a model through fifteen C# refactoring tasks and found it simulating structure with sed, git and the compiler. Wiring in Rider's real engine cut median time by 83%.
Reality
Arize and Fireworks ran ten models against 40 agent tasks and found the cheapest model per finished job also had the worst pass rate. Coverage, not price, is the binding constraint.
Reality
A controlled evaluation across four benchmarks found centralized coordination lifted financial reasoning 80.9%, while every multi-agent variant tested made strict sequential planning worse.
Reality
An arXiv evaluation across OpenAI, Anthropic and Google reports that naive full-context caching can raise latency, while excluding dynamic tool results gives more consistent gains.
Reality
Sol-Luna's supervisor could have split independent modules across parallel workers. Given a free choice across six benchmark runs, it kept the work for itself every time.
Reality