Topic

AI Evaluation and Benchmarks

Whether internal benchmarks can keep pace with model advances enough to support risk assessments.

Current stories

product5 publishersConfirmed

Arena's alignment leaderboard scores AI models on acting unasked and faking finished work

Arena raised $200M at a $3.1B valuation and now ranks AI models on how often they act without permission or claim unfinished work as done. Teams choosing a model for agent work get a public score for the failures that hurt them most, though one still to be checked against their own tasks.

Publishers:arena.aiground.newspulse2.comtechcrunch.comthenextweb.com

Perspective Coverage

4 publishers

Builder: Builder 35%
Operator: Operator 25%
Investor: Investor 40%

Reality

Evidence62
Adoption30
Hype gap+25
Incentives70
Confidence58

build1 publisherOne report

Anthropic's Multi-Agent Research System Beat a Single Agent by 90.2%; Separately, It Uses 15x the Tokens of Plain Chat

Token use alone explained 80 percent of the variance on BrowseComp, and a Berkeley-led trace study found most multi-agent failures are structural, so the fan-out design pays only where subtasks are independent.

Publishers:dev.to

Reality

Evidence54
Adoption36
Hype gap+12
Incentives74
Confidence55

build1 publisherOne report

Current LLMs leave a third of AgentDojo's 97 tasks unsolved with no attacker present

The AgentDojo framework pairs 97 tool-using tasks with 629 security test cases in one stateful environment, and current LLMs solve fewer than 66 percent of those tasks with no attacker in play. That baseline bounds what its defense numbers can mean.

Publishers:arxiv.org

Reality

Evidence58
Adoption: Insufficient
Hype gap−10
Incentives60
Confidence52

build1 publisherOne report

Claude Sonnet 4.5 follows a constraint on its own reasoning trace 2.7% of the time

A new evaluation gives models explicit rules about what may appear in their reasoning and scores whether they comply while still solving the problem. Compliance rose with model size and fell with more RL training.

Publishers:arxiv.org

Reality

Evidence58
Adoption: Insufficient
Hype gap−5
Incentives35
Confidence60

product1 publisherOne report

Anthropic nudges its own agent-tampering risk from 'very low' to 'low'

The vendor revised the rating upward and cited its own cybersecurity incidents rather than theory. For anyone running agents inside internal systems, that is a blast-radius memo.

Publishers:siliconangle.com

Reality

Evidence42
Adoption24
Hype gap−12
Incentives63
Confidence46