Topic

Benchmark Methodology and Reproducibility

Sensitivity of benchmark scores to prompts, temperature and agent scaffolding, and the methodological weaknesses of published benchmarks.

Current stories

build1 publisher

Cache lookahead brings a kaniko fork within 1.6x of BuildKit on GitLab runners

Kaniko's community fork, with cross-stage cache lookahead, ran 1.6x slower than BuildKit on GitLab.com runners in a dev.to benchmark. The 15x gap still quoted against kaniko dates from 2018, so teams choosing an in-cluster builder need figures from their own Dockerfiles.

Publishers:dev.to

Reality

Evidence40
Adoption: Insufficient
Hype gap+30
Incentives: Insufficient
Confidence35

product1 publisher

Qualcomm's own numbers give the Extreme Gen 6 a 44 percent GPU gain and 12 percent in single-core

Qualcomm has published first-party benchmarks for its new top Snapdragon tier, measured on its own reference hardware. The generational gain it reports depends almost entirely on which part of the chip a workload leans on.

Publishers:notebookcheck.net

Reality

Evidence38
Adoption12
Hype gap+32
Incentives82
Confidence58

build1 publisher

Which prompt built the test set decides Pangram's AI-assisted false-positive rate

Pangram's own 4.0 testing put the false-positive rate on AI-assisted documents at 0.01 percent, 4 percent or 7 percent depending on how the test text was produced, and the website shows the smallest of the three.

Publishers:lesswrong.com

Reality

Evidence48
Adoption30
Hype gap+20
Incentives60
Confidence42

build1 publisher

Five of eight DuckDuckGo MCP servers failed calls on the same twelve back-to-back queries

One machine sent the same twelve research questions through eight free servers, three times about an hour apart on 2026-09-15. The rule that caught most of the failures was the one for empty bodies.

Publishers:dev.to

Reality

Evidence62
Adoption52
Hype gap+10
Incentives65
Confidence58

build1 publisher

Two Incommensurable AI Code-Review Benchmarks Each Rank Their Own Product First

LinearB scored 16 hand-built bugs on noise and clarity; DeepSource ran the same product category against OpenSSF's public CVE corpus and published F1. Each vendor wins on its own instrument, and the two tests reward different behaviour.

Publishers:dev.to

Reality

Evidence42
Adoption: Insufficient
Hype gap+38
Incentives86
Confidence52

build1 publisher

HaloLog reconstructs span context before probing whether OpenTelemetry wants the record

The otelbridge adapter builds trace context first so the enablement check and the Emit call see the same values, because a processor downstream may filter on the sampled flag that a cheap probe would drop.

Publishers:dev.to

Reality

Evidence58
Adoption15
Hype gap−20
Incentives60
Confidence55

build1 publisher

Confidential inference on Blackwell retains 96-98 percent of throughput with CC-aware adaptations, NVIDIA reports

NVIDIA measured TensorRT LLM holding 96.1 to 98.2 percent of its non-confidential output throughput on Blackwell, and it got there by unpinning host memory on the affected paths, moving decode readback off the scheduler thread, and timing kernel tactics with the GPU's global timer instead of CUDA events.

Publishers:developer.nvidia.com

Reality

Evidence58
Adoption22
Hype gap+10
Incentives80
Confidence55

build1 publisher

Laravel's bootstrap cost 316KB of the 0.66MB a warm hello-world request used

A dev.to benchmark on Laravel 13.31.0 and PHP 8.4.22 splits a request into framework bootstrap and loaded data. The much-quoted 20MB to 30MB per-request figure turns out to measure PHP with OPcache disabled.

Publishers:dev.to

Reality

Evidence58
Adoption12
Hype gap+15
Incentives22
Confidence46

build1 publisher

Checking spline error between samples put 82% of tracks outside their 10 m tolerance

Fitting cubic B-splines to 200 GeoLife tracks looked like a search-accuracy problem until a dense error check showed the fitter had only ever validated at sample points, and the corrected fitter still lost to Douglas-Peucker.

Publishers:dev.to

Reality

Evidence60
Adoption8
Hype gap−12
Incentives18
Confidence58

build1 publisher

Saying "I authorize it" got past the CLAUDE.md rule in both runs that tried it

A 29-session test on Claude Code v2.1.273 ran the same protected-directory rule two ways, as prose in CLAUDE.md and as a PreToolUse hook. Both held on a plain task. The comparison that separates them rests on four runs.

Publishers:dev.to

Reality

Evidence58
Adoption: Insufficient
Hype gap+10
Incentives: Insufficient
Confidence55

build1 publisher

Word overlap in one listing line decided every skill invocation across 38 Claude Code runs

A test published on dev.to ran one commit-message skill through Claude Code 38 times, changing only how it was described, and found that "Helps with git stuff." never fired while a file with no frontmatter always did.

Publishers:dev.to

Reality

Evidence68
Adoption: Insufficient
Hype gap+8
Incentives30
Confidence62

build1 publisher

Martian scores code review bots on whether a developer actually changed the code

Code Review Bench reconstructs the timeline of 16,017 open source pull requests and publishes precision and recall beside every F1. The top five tools sit within five points of each other, on samples of very different size.

Publishers:dev.to

Reality

Evidence66
Adoption45
Hype gap+12
Incentives55
Confidence55

build1 publisher

Swapping the harness under a fixed model kept pass rate within 8 points on 50 Terminal-Bench Pro tasks

A 300-trial study swapped Goose, OpenCode and OpenHands-SDK under Qwen 3.6 Plus and MiniMax M2.5, and reports that the scaffold sets tokens per solved task and the failure mode while the score barely moves.

Publishers:arxiv.org

Reality

Evidence52
Adoption: Insufficient
Hype gap+20
Incentives30
Confidence58

build1 publisher

Per-vCPU steal sampling catches the unstable vCPU an aggregate average hides

A dev.to walkthrough for Linux guests on KVM splits a VDS benchmark into three separate measurements, records the environment before anything runs, and treats steal time as a correlation to test against throughput across repeated runs.

Publishers:dev.to

Reality

Evidence58
Adoption: Insufficient
Hype gap−8
Incentives25
Confidence55

build1 publisher

Five-field wait breakdown still computes connect and ttfb from the same formula

A dev.to harness stamps connect, TTFB, body, patch and test time on every generate call. In live mode the first two clocks come from one expression, so curl still names the handshake, and the published numbers are synthetic.

Publishers:dev.to

Reality

Evidence45
Adoption: Insufficient
Hype gap+55
Incentives70
Confidence68

build1 publisher

A whole-seconds cast zeroes the parse term in NVIDIA's JPEG2000 decode sample

The sample times the GPU decode correctly with CUDA events, then adds a Tier-2 parse term that a cast to whole seconds rounds to zero on every frame. Fastvideo puts that missing stage at 15 to 29% of decode time.

Publishers:dev.to

Reality

Evidence62
Adoption: Insufficient
Hype gap+14
Incentives78
Confidence55

product1 publisher

Fireworks' own DeepSWE numbers put four coding models inside the noise band

The vendor selling the cheapest model in the comparison reports a 0.7-point quality spread across four frontier models against run-to-run variation of 1.4 to 3.2 points. That leaves price per task, $0.43 against an implied $6.45 for GPT-6 Astra.

Publishers:fireworks.ai

Reality

Evidence42
Adoption18
Hype gap+28
Incentives88
Confidence58

build1 publisher

Mano-CUA's ten-point win over Claude shrinks to 0.8 points against Gemini

Mininglamp published NavEval scores for its own model on its own benchmark. Across the three entries, the spread tracks how each stack reads a page. It is not a case of specialists beating frontier models.

Publishers:dev.to

Reality

Evidence24
Adoption12
Hype gap+46
Incentives86
Confidence58

build1 publisher

On easy footage veryfast ties medium, but medium wins on size and quality for harder footage

Obole, an AI publishing one vertical video a day, ran libx264 settings four times each on a near-static clip and on a synthetic noise clip. On the easy file, -preset slow bought 0.16 percent of the bytes for three to five extra seconds.

Publishers:dev.to

Reality

Evidence70
Adoption15
Hype gap−8
Incentives25
Confidence62

build1 publisher

A bare except handler hid thousands of unreadable tables behind an 86.1 percent recall score

The loader read a Spider 2.0 checkout with rglob and swallowed every read error, so 2,868 files Windows would not open became tables that do not exist, and the benchmark scored recall against what was left.

Publishers:dev.to

Reality

Evidence50
Adoption: Insufficient
Hype gap+5
Incentives25
Confidence55

Earlier coverage

Postgres walks all 110,659 pages of a 1.2 GB table to answer OFFSET 9999980 LIMIT 20
Build · September 15, 2026 · 1 publisher
Trail of Bits says 1Password's 26% AI patch score reflects flawed prompts, no-code-execution trials, and grading errors, not true AI performance
Security · September 15, 2026 · 1 publisher
Rerunning the same eval suite three times in ten minutes moved its score by two cases
Build · September 14, 2026 · 1 publisher
AllSpark ran each Iris benchmark twice to separate the model from its scaffolding
Build · September 13, 2026 · 1 publisher
Dropping a 6.6-second trace phase saved more build time than Turbopack's faster compile
Build · September 11, 2026 · 1 publisher
One predicate removed 48.5% of a test operator's steady-state reconciles without weakening repair
Build · September 7, 2026 · 1 publisher
A token-overlap matcher shrugged at more than half of 1,538 agent rule candidates
Build · September 7, 2026 · 1 publisher
Speed and archive size rank the same nine backup plugins in opposing orders
Build · September 7, 2026 · 1 publisher
Measuring each patch against the human fix on the same bug leaves 13 of 14 models messier
Build · September 6, 2026 · 1 publisher
A .NET 10 lab benchmarks four publish modes against ten metrics on one laptop
Build · September 4, 2026 · 1 publisher
Twenty passing runs move the coding-API decision onto time-to-first-token
Build · September 2, 2026 · 1 publisher
Reordering two loop indices beat every cache-tiled version on the same machine
Build · September 2, 2026 · 1 publisher
Meta prices streaming transcription at a fifth of Google Cloud's standard rate
Leadership · September 1, 2026 · 1 publisher
PgCache invalidates cached aggregates off Postgres's logical replication stream
Build · September 1, 2026 · 1 publisher
fal reports 35x MiniMax's own H3 endpoint after tuning the weights to its runtime
Build · September 1, 2026 · 1 publisher
The choice of speaker encoder moved equal error rate five-fold on an identical trial list
Build · August 31, 2026 · 1 publisher
Farm.js compiles a state update into a direct DOM write when it can prove the target
Build · August 31, 2026 · 1 publisher
LatticeDB's traversal speed margin over SQLite narrows by three hops, then widens sharply by depth ten
Build · August 30, 2026 · 1 publisher
The repo's own control run deleted the 5-10x WASM claim from vizcrush's launch copy
Build · August 29, 2026 · 1 publisher
Eleven agent sessions on one machine settled CPU contention by writing to each other
Build · August 28, 2026 · 1 publisher
WebKit charges 39 MB for the tab Kestrel's simulator priced at 32 KB
Build · August 28, 2026 · 1 publisher
Harness choice moved token use 83-fold with the model held constant
Build · August 27, 2026 · 1 publisher
The one Pylint check Ruff cannot take, and the case for pulling it out of CI
Build · August 24, 2026 · 1 publisher
Optuna killed 60% of the trials and gave back 28% of the time. The gap is in the config.
Build · August 24, 2026 · 1 publisher
Harness-Bench makes the scaffold a measured variable, and model-only scores lose meaning
Build · August 23, 2026 · 1 publisher
A 284B model at 25 tokens a second on one 5090, and 192 GiB of DDR5 doing the quiet part
Build · August 22, 2026 · 1 publisher
Uniform INT4 beat NVFP4 in 85 of 90 real gradient tests, and the rotation barely moved them
Build · August 22, 2026 · 1 publisher
AgentCL: if the task stream is not controlled, agent memory gains prove nothing
Build · August 21, 2026 · 1 publisher
A five-way graph database benchmark spent its first four hours measuring undersea cable
Build · August 21, 2026 · 1 publisher
Tokenize-then-compress works, but the win is window coverage, not the 45% smaller stream
Build · August 20, 2026 · 1 publisher
The 21-cent model bake-off that inverted when the judge got audited
Build · August 20, 2026 · 1 publisher
A JSON parser benchmark that scores refusal as a pass, and why the column order flips
Build · August 19, 2026 · 1 publisher
Next.js 16.3's memory claim didn't reproduce; its TypeScript handoff cut a build by two thirds
Build · August 18, 2026 · 1 publisher
Your GPU reports 24GB. Only 7.9GB of it loads a model, and half of that is already gone
Build · August 18, 2026 · 1 publisher
A harness gain is not a leaderboard win: reading the J-Space DeepSeek report properly
Build · August 17, 2026 · 1 publisher
Artificial Analysis moves eval onto your data, and turns model choice into procurement
Build · August 15, 2026 · 1 publisher