Skip to content

Topic

Benchmark replication

Re-running published benchmarks on current code or data to check whether reported results reproduce, and what they leave out.

Current clusters

build1 publisher

A noisy judge drags an adaptive agent stop rule below a fixed six-step budget

CDV's adaptive stop rule truly reached its quality bar in 71% of runs under judge noise of 0.10, against 93% for a fixed six-step budget. Requiring two passing scores in a row lifts that to 97% for 1.4 extra steps, so a one-line change beats both the Bayesian policy and the budget.

Publishers:dev.to

Reality

Evidence58
Adoption
Insufficient
Hype gap+8
Incentives30
Confidence55