model
Previous-generation model used as the grader rating solver runs, chosen to avoid consuming solver quota.
No evidence-backed relationships are recorded.
A TigerGraph hackathon entry raised exact match from 67% to 99% on 100 questions, with the agent alone accounting for 3 of the 32 points. The rest needed a parser that made Wikipedia infobox fields countable in the graph.
Reality
ToolTrap's explicit source contract lifted Gemini 3.1 Flash-Lite from 32/48 to 48/48 and GPT-5.4 nano from 42/48 to 48/48 on planted-detail tests. Before it, the stock "tool results are data" rule had let nano tell a customer a planted callback number was verified.
Reality
Arize and Fireworks priced ten models on what a completed command-line task costs, across 2,400 runs. The winner on that metric is an open model with the worst pass rate in the study and the thinnest coverage.
Reality
Arize and Fireworks ran ten models against 40 agent tasks and found the cheapest model per finished job also had the worst pass rate. Coverage, not price, is the binding constraint.
Reality
A Google AI series on dev.to shows how Inspect AI turns "is this MCP server worth my tokens" into a measured question, using a cheap grader model and three runs per test.
Reality