Topic
Designing evaluations that resist training-data contamination and measure research quality, including the Shadow Evaluation protocol and critiques of peer review as a benchmark.
A dev.to post gives every failed coding-agent run one of four labels and scores skill only over the runs where the harness stayed healthy. SSH drops and disk-full errors get published as their own rates.
Reality
Datadog Security Labs put one document-portal prompt through three coding agents in both modes and audited all six builds. An insecure direct object reference appeared in each, and one build per cell leaves mode effects entangled with noise.
Reality
A replay that held the agent's actions fixed produced different labels before and after delayed operations resolved, and one late write moved the following run's score.
Reality
A self-published log of 157 agent runs says planning depth beats model size. Model size was never one of the dimensions the runs varied.
Reality
One good demo is not evidence that a new prompt, skill or rules file helped. A control arm, thirty tasks and a held-constant model will tell you, within limits worth knowing before you build it.
Reality
A preprint from ServiceNow AI Research and university co-authors reports that a plain web agent with a longer horizon matches or beats AWM, ASI and ReasoningBank, often on fewer tokens.
Reality
A proxy on the MCP stdio pipe caught one client killing most of its own tool calls before they reached the server. On the wire it looked exactly like a weak model.
Reality
One research agent, built three times on three hyperscaler frameworks with the same prompt and the same search tool, diverged in nine ways. None of them were protocol problems.
Reality
A thirty-run experiment finds agents followed long, buried rules anyway. The two failures came where the file contradicted the repository, not where it was long or buried.
Reality
Princeton and the UK AI Security Institute gave a frontier agent six days and $3,000 to answer real unpublished research questions. The original authors reviewed the output and rejected both papers.
Reality