Epoch AI's benchmark audit fails nine of the first 15 versions it inspected

The defects sit in answer keys and graders. Epoch AI's review of Berkeley Function Calling Leaderboard v4, which tests whether a model selects the right tool and supplies the right arguments ^[12], found a task that expected the model to add a stock already present on the watchlist ^[13]. A model that checks the list first and declines the duplicate is scored as a failure. Another task's web-search answer had become stale since the key was written ^[13].

In Humanity's Last Exam, one question asked for a four-point discrete Fourier transform and supplied eight entries ^[10]. Another answer key pointed at the wrong multiple-choice option ^[10]. Epoch AI reported problems that could reject valid answers and problems that could accept incorrect ones ^[11].

The sampling rules explain the sample sizes. Epoch AI takes 50 tasks from a benchmark with more than 50 available, expands to 100 when the observed error rate falls between 15 and 25 percent, and inspects every task when there are 50 or fewer ^[6]. DeepSWE v1.1's confirmed false negative rate is 20.3 percent ^[14], inside that band, and the 113 tasks inspected are past the expansion level ^[4]. Humanity's Last Exam at 46 percent and Berkeley Function Calling Leaderboard v4 at 48 percent are both above the band ^[8][2], so a 50-task sample settles them ^[5]. The Humanity's Last Exam sample was 48, two short of the stated 50, which at that error rate does not change the verdict ^[8].

Epoch AI stops a review once it has gathered enough evidence for a Flawed verdict, and says the writeups should be read as a minimum case ^[7]. A Flawed label bounds the defect count from below.

Nine Flawed out of 15 is 60 percent ^[1]. The registry Epoch AI maintains holds 85 benchmarks ^[15], so the first batch covers at most about a fifth of it ^[3], and the launch material does not say how the 15 versions were picked. For that 60 percent to transfer to whatever benchmark you are about to cite in a deployment review, the selection would have to have been blind to which tests already looked suspect. The per-benchmark findings hold on their own.

The reviews were completed between August 10th and September 12th ^[16], so the September 17th launch published an accumulated program. Epoch AI, co-founded and led by Jaime Sevilla, argues that AI progress needs better measurement than leaderboards and launch-day claims supply ^[18]. Epoch AI says its founding group was frustrated that a consequential industry was still being measured through "hype and vibes" ^[17].