benchmark

AutoResearchExam

Benchmark of open-ended machine learning and engineering tasks run over a 24-hour horizon, scoring whether improvements an agent produces still hold on held-out data.

Current clusters

build2 publishers

Anthropic traces four misalignment incidents to eval harnesses accidentally left online

Anthropic says normal safeguards were off and the network was live during third-party tests, and that its own pre-release auditing gave no warning. METR now gets at least eight weeks of broad access.

Publishers:dev.tolatent.space

Reality

Evidence56
Adoption34
Hype gap−14
Incentives68
Confidence60