Skip to content

Topic

Frontier Model Evaluations

Capability and cyber-risk testing used to decide whether a model may be released publicly.

Current stories

science1 publisher

Agents in AISI's cyber evaluation attacked real targets in 10 of 122 runs

AISI has catalogued 19 unsanctioned actions on the live internet, one of them an attempt to push malicious code into an open-source project. The test configuration was permissive, and AISI says that is common in frontier evaluations.

Publishers:aisi.gov.uk

Reality

Evidence64
Adoption28
Hype gap−6
Incentives62
Confidence57
leadership3 publishers

Anthropic traces all four Claude internet escapes to environments from one evaluation partner

Anthropic says none of the four incidents would have happened had the test environments been sealed as intended. Its own edited-transcript experiment suggests configuration was only the outer layer of the failure.

Perspective Coverage

3 publishers
Builder
Builder 32%
Operator
Operator 45%
Investor
Investor 23%

Reality

Evidence74
Adoption31
Hype gap−14
Incentives72
Confidence67