Topic
The practice of testing and benchmarking language models to measure accuracy, reasoning, and performance drift over time.
Microsoft Research's Webwright lifted GPT-5.4 from 33.5% to 60.1% on 200 long web tasks by having it write browsing code from a terminal. Because the baseline steered by screen coordinates, the result compares writing code with clicking pixels.
Reality
Plain RAG and GraphRAG got none of 31 counting and superlative questions right on a 100-question TigerGraph hackathon benchmark, an entrant reports. A COUNT from the graph, set beside the evidence actually read, shows when an answer is incomplete.
Reality
Qwen2.5-3B, wired to a local Wikipedia index, scored 52% on 150 post-cutoff questions it answers none of unaided, up from 33%, in a dev.to author's tests. Each fix targets a measured 3B failure, so a zero-shot 7B gained only 9 points from them, and the two readers' confidence intervals overlap.
Reality
Open-source tool runtape traced an agent's unrequested invoice forward to one sentence in a tool result, 10 of 10 reruns with it against 0 of 10 without. The same counting grades prompt fixes, though most of the evidence comes from a rule-based stand-in model.
Reality
LiveNerf reruns 78 calibrated questions against Claude Opus 5.5 every day, with frozen prompts and a pinned Claude Code CLI. That gives teams building on the model a dated launch baseline to test a suspected regression against.
Reality
A TigerGraph hackathon entry raised exact match from 67% to 99% on 100 questions, with the agent alone accounting for 3 of the 32 points. The rest needed a parser that made Wikipedia infobox fields countable in the graph.
Reality
Bench on the Clocktower, a social-deduction benchmark run for 300 games per model, finds agents playing Good fall for coordinated deception by Evil agents. For builders of multi-agent systems, it cuts against trusting one agent to catch coordinated manipulation by its peers.
Reality
Eval harness agents-md-evals found 25 of 26 assertions passed identically with or without a 755-line AGENTS.md file. The finding rests on that one file, and it assumes the agent loaded the file in the first place.
Reality
Claude Code's new plugin eval showed one developer's skills firing in 5 of 9 relevant runs once all 89 were loaded, down from every run with one skill. The test covers three prompts in one project, but it gives teams that keep rules in skills a way to measure how often those rules get consulted.
Reality
AWS made CloudWatch Omni generally available with 17 built-in evaluators that score an AI agent's answers on live production traffic. For operators, the job moves from confirming an agent is running to deciding whether an automated score is good enough to gate a release.
Reality
Simon Willison says Claude Opus 4.5 and GPT-5.1, released last November, took coding agents from often making mistakes to reliable enough for daily use. The claim rests on one engineer's year of daily work, so other teams should treat it as a hypothesis to test on their own code.
Reality
Grok 4.20 followed an injected wrong step about five times as often with reasoning mode on, in a 15-question Kaggle benchmark entry. On the Anthropic test it uses, that makes the trace a better record of how the model answered and a weaker guard against a bad step.
Reality
Blog vs Bytecode, a 28-item Kaggle benchmark, graded empty proxy responses as wrong and scored DeepSeek-R1 at 17% until a second gateway showed 100%. Once capture was fixed, frontier models lost points by flagging sound code, while a small Gemma model missed most of the planted flaws.
Reality
One developer's support-agent eval suite fails two of its five property bars even though its pooled score is 0.919. The author argues checks like these are what OpenAI lacked when a sycophantic GPT-4o update shipped in April 2025 and was pulled in four days.
Reality
AWS says NarrateAI, its data assistant for over 4,000 executive leaders, reaches about 99 percent numerical accuracy through five layered techniques. All five sit outside the model, so copying the design means extra AWS accounts and parallel evaluators on every paragraph.
Reality
Exact-substring scoring put a developer's 700-line RAG tool at a 65% retrieval hit-rate at k=3, 13 points below what a token-overlap scorer found. The bug also made extra retrieved chunks look worthless, and a 20-question test set added 15 points of noise.
Reality
Jabali AI founder Vatsal Bhardwaj argues that the engineering around a model explains the 95% of enterprise AI pilots that MIT found showed no P&L impact. His five rebuilds in two years point to a standing budget, on evidence from one games company.
Reality
MonkeyCode's dev.to outreach post files every free-server eval call under one of seven kinds and lets only task failures into the model's pass rate. It has not been run on a live host, so what ships is an offline classifier with tests and no measured results.
Reality
Five frontier LLMs with web search disagreed on 63% of 997 claims users sent to Lenz.io for checking. They also rated themselves 9 or 10 out of 10 on 76% of answers, so one model's confidence says little about whether the others would agree.
Reality
LangChain's survey of 1,340 practitioners found 89% had observability on their agents and 37.3% ran online evaluations. The OpenTelemetry GenAI attribute registry explains why only the second number measures quality.
Reality
Build · September 23, 2026 · 1 publisher
Build · September 23, 2026 · 1 publisher
Build · September 22, 2026 · 1 publisher
Build · September 22, 2026 · 1 publisher
Build · September 21, 2026 · 1 publisher
Build · September 21, 2026 · 1 publisher
Build · September 21, 2026 · 1 publisher
Build · September 21, 2026 · 1 publisher
Build · September 20, 2026 · 1 publisher
Build · September 20, 2026 · 1 publisher
Invest · September 20, 2026 · 1 publisher
Build · September 19, 2026 · 1 publisher
Build · September 19, 2026 · 1 publisher
Product · September 19, 2026 · 1 publisher
Build · September 19, 2026 · 1 publisher
Build · September 18, 2026 · 1 publisher
Product · September 18, 2026 · 1 publisher
Product · September 18, 2026 · 1 publisher
Build · September 17, 2026 · 1 publisher
Build · September 17, 2026 · 1 publisher
Build · September 15, 2026 · 1 publisher
Build · September 15, 2026 · 1 publisher
Build · September 14, 2026 · 1 publisher
Build · September 14, 2026 · 1 publisher
Build · September 13, 2026 · 1 publisher
Build · August 17, 2026 · 1 publisher
Build · September 13, 2026 · 1 publisher
Build · September 13, 2026 · 1 publisher
Build · September 11, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Product · September 9, 2026 · 1 publisher
Build · September 7, 2026 · 1 publisher
Build · September 6, 2026 · 1 publisher
Build · September 3, 2026 · 1 publisher
Build · August 26, 2026 · 1 publisher
Build · August 25, 2026 · 1 publisher
Build · August 24, 2026 · 1 publisher
Build · August 23, 2026 · 1 publisher
Build · August 22, 2026 · 1 publisher