Reproducing 2,200 ICML papers: 23% contested, and 242 of those contested each other

Hugging Face has published results from a three-week hackathon in which community teams pointed coding agents at ICML 2026 papers and tried to reproduce them, roughly 2,200 in total ^[1][10]. The organizers' argument is that AI-accelerated submission volume has outrun review capacity, and that the same agents driving the flood can be turned around to check it ^[5][6][9]; the numbers support the first half more cleanly than the second.

The volume case is straightforward. ICML 2026 received 23,918 submissions and accepted 6,352 papers, roughly double the previous year, which the organizers attribute at least partly to agents making experiments and write-ups faster ^[3][4][5]. That is an acceptance rate near 26.6% ^[1]. Reviewing capacity did not double, and reviewers remain volunteers who may lack the time or expertise to fully check a paper ^[6][7]. The post quotes a reviewer of an accepted ICML 2026 spotlight: "My low confidence score is because I did not check all the proofs carefully" ^[8].

The mechanics were deliberately cheap. The challenge ran July 15 to August 2, 2026, gave each participant $20 in Hugging Face compute credits, and produced 2,962 cloud jobs ^[10][11][12]. Where datasets were proprietary or checkpoints unreleased, participants ran toy reproductions on synthetic data with similar properties ^[13].

Aggregated per paper: 1,103 papers, 51% of those examined, had at least one claim independently verified, including 266 fully reproduced and 632 partially reproduced with nothing falsified, on 3,978 confirmed claims ^{[14][15][16][17]}. That is about 3.6 confirmed claims per verified paper ^[5], and the 51% figure implies roughly 2,163 papers examined ^[2]. Against that base, full reproduction ran at about 12% ^[4].

The failure side is where the method shows its limits. 496 papers, 23%, had at least one claim falsified or contested, including 49 where everything was falsified and nothing verified ^[18][19]. But 242 papers had independent teams reaching opposite verdicts on the same claims ^[20], which is 49% of the contested set ^[6]. Add the 502 papers with toy-scale evidence only and the 280 where nothing could be established either way, mostly for missing artifacts ^[21][22], and 782 papers, about 36% of the examined set, yielded no usable signal ^[3]. A $20 budget buys a lot of parallelism and not much resolution.

What actually closed the loop was human labor. 35 participants formally claimed a falsification, and the organizers adversarially re-verified every one by re-reading the paper and the logbook and re-deriving the math or re-implementing the experiment from the paper's own text ^[23][24]. The headline case is the paper the unconfident reviewer had reviewed: "Towards Optimal Robustness in Learning-Augmented Paging," where a participant's logbook found the additive term in the robustness bound growing faster than claimed and identified the exact step of the proof that breaks ^[25][26]. The organizers extended the sweep to k = 1,024 and confirmed the growth at roughly nine sigma ^[27].

So the honest reading is narrower than "agents replace reviewers." Agents generated 2,962 job runs and a large pile of ambiguous verdicts; a small number of humans turned one of them into a specific, located error in a spotlight proof ^[12][20][26]. Watch whether the 242 disagreements get adjudicated, since an unresolved contradiction is a claim about tooling, not about the paper. Watch whether the 280 papers with missing artifacts move conferences toward mandatory artifact release ^[22]. And watch whether ICML 2027 submissions double again.