Topic
Self-published benchmark tables, private evaluation sets and the absence of independent reproduction as a verification problem.
A dev.to post lays out a coding-agent eval protocol that hashes the token and tool-call envelope alongside the tasks, so editing a cap changes the run id. No executed runs are published with it.
Reality
The designated successor to GenAI-Perf splits load generation from record processing so the benchmark client stops hitting Python's GIL. Teams holding a GenAI-Perf baseline inherit a port and a re-run.
Reality
A dev.to post publishes a TypeScript decorator that records every cache key written for an entity, so invalidation reads a short list instead of walking the keyspace, and it ships the benchmark scripts because every figure in it is a local run.
Reality
The Neve author has published Frost with a runnable ResNet-18 benchmark and three named complaints about PyTorch. The 1,400-line count covers the framework layer only; the compiler and SIMD backend underneath are separate code.
Reality
The published doubling was measured with a per-task allowance most teams will never grant, under a harness Anthropic has not described. The number that transfers is the one you get at your own spend cap.
Reality
Publishing weights and publishing something a team can actually deploy are different acts, and N2.5 lands on both sides of that line depending on which tier you pick and whether its weights exist yet.
Reality
The Accio team's verifier ignores what the agent says it did and inspects the containers it leaves behind. That design is the news; the two-pass open-weight lead Alibaba reports from it sits inside its own harness spread.
Reality
Six self-reported write-ups on stacking, voting, blending and search kept landing inside their own seed spread. The clearest result was procedural: in-fold stacking weighted the worse model 32 to 1.
Reality
ArmBench-ASR v0.1 ranks nearly 30 systems on 20.7 hours of Armenian audio. The headline order flips on read speech, and every model degrades badly on movie dialogue.
Reality
A Google AI series on dev.to shows how Inspect AI turns "is this MCP server worth my tokens" into a measured question, using a cheap grader model and three runs per test.
Reality
Z.ai's August 14 post claims post-training gains for coding agents, but the company's release notes still stop at GLM-5.1 and there is no API endpoint, model identifier or weight download.
Reality