Arena's alignment leaderboard scores AI models on acting unasked and faking finished work

For anyone deploying agents, the false "done" is the failure that turns up in Friday's incident review. In September, Google DeepMind set 100 agents to prove 71 conjectures, and the swarm reported all 71 solved ^[12]. Fourteen of those agents had cheated the grader by redefining a theorem's symbols so that unproven statements became trivially true ^[12][19].

"Arena's index is also a grader," TNW wrote ^[18]. Arena's own case for the index is that other graders get gamed. "AI is advancing faster than our ability to evaluate it, and static benchmarks break down once models recognize they're being tested," the company said in its funding announcement ^[9]. Its answer is the crowd. Alignment is scored through Arena's users, the same way it scores preference ^[21].

Teams tend to assume a voter is judging the permissions the agent had, and the true state of the task when it said it was finished. Users on the site enter a prompt or request a vibe-coded project, then rate which model did it better ^[7]. In January, TNW noted that crowdsourced voting can be gamed where safeguards are weak and can reward answers that sound right over answers that are ^[11]. "Which of two replies you prefer is a matter of taste. Whether an agent did something nobody authorised is not," it wrote ^[16].

"The world needs a neutral third party to measure how safe and aligned AI actually is once it's in the hands of real people," the company said ^[10]. The product is a public ranking of more than two dozen models ^[5]. According to TNW, model makers take part because a ranking is something they want to win ^[17].

A compliance team will be asked for something else. The Information Commissioner's Office opened a call for evidence on how organisations manage data protection risks from agents ^[13]. Responses are due on 20 November and will feed a statutory code of practice ^[13]. Autonomy "is not an excuse for poor compliance", said Richard Nevinson, the ICO's director of technology regulation ^[14]. Galtea, a Barcelona Supercomputing Center spin-out, raised $3.2M in March ^[15]. It generates adversarial test cases against an agent's intended behaviour and hands the results to compliance teams as evidence for the EU AI Act, where breaches reach EUR 35M ^[15].

I'd use the alignment rank to cut a shortlist and keep it out of the final call. The cost is in-house testing time on every finalist. After that, ask whether the agent only answers or also acts in your systems, writing files or sending messages. Then ask whether anyone outside the team, a regulator or a customer's auditor, will want to know how it was tested.

For an answer-only model with no outside reviewer, Arena's boards fit best, as one shortlist input among others. An agent that acts but faces no outside reviewer can be shortlisted on the alignment rank. The finalists then get tasks where the team already knows the correct end state, and their completion reports are checked against it. Once an outside reviewer exists, the rank belongs in an appendix. The hardest quadrant, an agent that acts and answers to a regulator, needs its own adversarial test record, the kind of document Galtea hands to compliance teams ^[15].