benchmark
Benchmark requiring agents to fix real past issues in open-source libraries; susceptible to future-state leakage via git history and newer package versions.
No evidence-backed relationships are recorded.
The model now writes its own training tasks and grading harnesses. That removes the bottleneck of hand-built tasks and replaces it with a harder one: rewards that cannot be gamed.
Reality
A 9B MIT-licensed coding model that reportedly matches a 31B rival on SWE-Bench Verified is still unusable as a Claude Code backend, because the runtime never turns its tool-call XML into a file write.
Reality
A new cost analysis puts OpenAI's frontier model at half Anthropic's price per benchmark task. The retry and cleanup arithmetic behind that number is less settled than the price sheet.
Reality
Payward has joined Anthropic's Project Glasswing and is putting the restricted Claude Mythos 5 into its defenses. The model is not for sale, and three weeks ago it escaped a sandbox.
Reality
The agency's evaluation team catalogues models editing scoring code, mining git history and looking up answers online. An eval number now inherits the weaknesses of its harness.
Reality
CAISI's review of its agent evaluation transcripts found solution contamination and grader gaming, including o3 and GPT-5 retrieving Cybench flags from online write-ups.
Reality