Topic
Sensitivity of benchmark scores to prompts, temperature and agent scaffolding, and the methodological weaknesses of published benchmarks.
A 452-configuration benchmark of the parmar pre-filter splits two effects that looked like one: denser tokens buy a flat 15% on gzip, while the gain that scales comes from window expansion.
Reality
Five models, ten questions, a tidy leaderboard. Then the author checked who was grading, found a contestant holding the pen, and re-scored the saved answers for three cents.
Reality
Seven Python parsers, 300 labelled malformed LLM outputs. On recoverable cases json-repair wins by two; count the 25 unrecoverable ones and it loses by seventeen.
Reality
One production app, taken from 16.2.3 to 16.3.1 with readings recorded on both sides, is the first published independent test of Vercel's "90% less RAM" figure. The useful win came from elsewhere.
Reality
A single 8GB laptop produced four different VRAM figures, none of them a bug. The one that decides whether a model loads is the one no tool puts in front of you.
Reality
A viral X post said an inference-time text layer put DeepSeek V4 Pro ahead of Fable 5 on every task. The report it points to shows single runs, nine benchmarks, and two losses.
Reality
Optima lets buyers build benchmarks from their own datasets and agent traces, then scores candidate models on quality, cost per task and time per task.
Reality