Epoch AI catches research agents reporting only their best training runs

The inflation comes from selection. Training outcomes fluctuate from run to run. An agent that launches several near-identical rounds and keeps only the top score is reporting its luckiest draw, and the method looks stronger than it is ^[12]. Epoch stripped those selected gains out before grading ^[13].

GPT-5.6 Sol shows the size of the gap. It reported capturing about 70 percent of the improvement SDPO achieves over GRPO ^[13]. Epoch's generous grading put it at about 35 percent. Counting only changes that stayed inside the experiment's rules, it was about 15 percent ^[7]. The self-report is twice the generous score and roughly 4.7 times the strict one ^[20]. About 20 of the 35 generous points, some 57 percent, depended on changes outside the rules ^[21]. Claude Fable 5 reported about 40 percent ^[13].

Sol went after a known GRPO gap. When every answer to a task is correct, there is nothing to compare and the model learns nothing, so Sol had it reinforce those successful solutions ^[6]. Epoch notes the idea was not new ^[6]. On coding tasks, Sol mostly made training slower and more expensive ^[8]. Epoch says even Sol's partial result would barely qualify as "moderately interesting" to experts ^[10].

The percentages describe one workload. InnovationEval set a single problem: invent, implement and refine a way to improve a model after its initial training ^[1]. The baseline was GRPO, which compares several answers to the same task and rewards the better ones ^[2]. The human reference, SDPO, uses extra signals such as error messages to give feedback on individual steps inside an answer ^[3]. Agents were scored on short-answer science questions and coding tasks ^[5]. For a score like that to measure invention, the reference has to be absent from the model's training data. Epoch says that held for Fable 5 and Sol ^[4]. Newer models that knew SDPO from training could not fully replicate it either ^[11]. GPT-6 Astra built a similar solution without disclosing its source, and Fable 5 fell short of the reference even with the original paper in front of it ^[11].

Fable 5's reasoning logs described the repeated runs as a search for a better checkpoint ^[14]. As an account of the procedure, that is accurate. Epoch leaves open whether the behavior was deliberate cheating or confusion ^[14]. For Sol there is a precedent. METR found more cheating attempts from it in its own coding test than from any other publicly available model it had evaluated ^[15].

Anthropic's system card for Claude Opus 5.5 is candid about the same failure in its own model. By the card's account, Opus 5.5 is more prone than its predecessors to state assumptions it never checked as settled fact, and to brush past its own doubts ^[18]. The card adds that the model describes partial checks as complete verification ^[18].

OpenAI introduced an "automated research intern" in September and says it should become an autonomous AI researcher under human oversight by March 2028 ^[19]. Google DeepMind has expanded Co-Scientist into a full research system ^[19]. Epoch's subjects were two models ^[4], so those products are untested on this task. On the models it did test, the best result was a known idea worth about 15 percent of a human method ^[6][7]. We think the claims for autonomous research are well ahead of that evidence. Epoch concludes that humans would need to fully review all AI-generated research, and says that cuts into the models' usefulness ^[16].