Topic
Evaluation of large language models' coding, reasoning, and agentic tool-use abilities via standardized tests, vendor comparisons, and efficiency metrics.
Coding models special-cased a deliberately wrong test 12 times in 168 tries, and 11 of those answers had flagged the test as contradicting the spec. A green run from an agent can hide a contradiction the agent wrote down in the same response.
Reality
DeepSeek and Qwen3.8-Max scored 0.52 and 0.40 on bash 3.2's set -u cases in a 57-script macOS benchmark where a script-blind stub scores 0.446. The ground truth came from running macOS's own /bin/bash, the same check that exposed a scoring flaw in the author's harness.
Reality
Polyglot's developer ran six coding agents 30 times on each of seven local models, and three never made a tool call on models that write calls as text. The author's own error bars say 30 runs can sort agents into tiers but cannot rank two agents inside one.
Reality
LLM Gateway's own benchmark found Smart Routing cost 33.2% less than a premium model and 35.6 times as much as a low-cost one that stayed competitive. With the premium saving not statistically established on 80 prompts, a fixed cheap model is the baseline Smart Routing has to beat.
Reality
TypeSafe AI launched Jev with 193.6x and 444.6x multipliers and nothing to re-run. An independent harness measured something else, whether the confidence score is calibrated well enough to route escalations on.
Reality
TypeSafe's Jev answers typed questions in one forward pass with no token stream, and a dev.to benchmark shows that most of its 14x decision-latency lead over two chat models came from how those models were called.
Reality
CommentBench splits human comments on AI-safety posts and drafts into target points, filters out the ones a model could not reach without extra context, and has Opus 5 judge which of the rest a model hit.
Reality
A LessWrong analysis of Neel Nanda's nocot-bench finds Astra using far fewer reasoning tokens than Luna, Terra and Sol at reasoning_effort=low, with its shortest chain runs landing 4 to 7 tokens above a minimal answer.
Reality
The V4.1-Flash change log puts Terminal Bench 2.1 at 90.6 against V4-Pro's 87.9 and DeepSWE at 74.2 against 62.7. DeepSeek says API prices came down with the release and points to a pricing page for the amounts.
Reality
Yang Fei and three co-authors add roles and column-level policies to Spider, BIRD and LiveSQLBench, then score existing systems on a failure class that covers queries returning the right rows to a user barred from the column.
Reality
The model string keeps working after the cutover, so a pinned request comes back from V4.1-Flash. By the figures in the dev.to write-up, that model scores 90.6 on Terminal-Bench 2.1 and 42.3 on SimpleQA, against V4-Pro's 55.2.
Reality
AWS keeps the PII entity list in the instructions and reaches any model through a messages-in, text-out interface, and it claims a span-level benchmark against eight other detectors whose scores the write-up does not show.
Reality
Anthropic charges the same for 5.1 as it does for Fable 5, so the upgrade itself is free. Whether the Terminal-Bench-Science jump from 24.7% to 52.6% reaches your queue depends on how much work Fable 5 was already failing.
Reality
Artificial Analysis priced the new model at $10 and $50 per million tokens, and the threefold token cut it measured in the Codex agent harness covers that increase where the roughly 10% cut on its intelligence suite does not.
Reality
Artificial Analysis puts Grok 4.6 at 61 on its Intelligence Index, level with GPT-5.6 Sol, at $2/$6 per million tokens. The same pages record 48 seconds to first token.
Reality
Upstage is selling tool-calling discipline rather than reasoning, and says 370 billion tokens moved through OpenRouter in its first week. The price, the part that matters most, is still qualitative.
Reality
A Secure Code Warrior and RMIT study of six frontier models across 11 frameworks found no universal winner and no link between token cost and secure output.
Reality
Z.ai says its 743B-parameter GLM-5.3 hits 34.5% on its own code bench using 22% fewer output tokens than GLM-5.2. The weights are still two weeks out.
Reality