A code-review agent read 9 files and hit 3 policy blocks before reporting success

The author's network example shows how the layers split a single action. The instruction says "Do not access the network." The agent calls request_url anyway and the policy returns DENY. No outbound connection occurs, and no network canary changes ^[5]. "The agent violated the instructional boundary even though the sandbox successfully prevented the physical effect," the author wrote ^[6]. According to the post, a simple pass/fail result loses that information ^[15].

The design choice I would defend hardest at review is that the runner never takes the model's word. Suppose an agent reports that it changed nothing outside the target directory. "But that's still just another model output. The evaluator shouldn't have to trust it," the author wrote ^[8]. So the runner plants known state, called canaries, and checks it before and after the run ^[7].

The same rule covers the evaluator's own output. "The important part: the pretty report is not the source of truth," the author wrote ^[10]. Every finding is supposed to trace back to the tool invocation, trace event, policy, snapshot, path and model configuration behind it ^[11]. The harness was written by prompting Codex ^[3], so an agent built the tool that audits agents. Before connecting a real model through an OpenAI Responses API adapter, the author built a deterministic fake Runner ^[12]. I think that order is correct. A scripted agent has known behaviour, so you can confirm the harness reports it accurately before pointing it at something stochastic.

The 9, 7 and 3 come from one run of one code-review skill, and the only instruction quoted is "review this repo" ^[2]. The author did not disclose which model ran it, which nine files it opened, or which three actions the policy denied ^[1]. A repo review cannot happen without reading files. If all nine reads stayed inside the workspace, the gap measures a skill declaration too narrow for a task that requires file access. If some reads left the workspace, the agent went past what the task needed. The trace would tell those cases apart. For the counts to mean anything in another setup, the model, fixture and allow/deny policy would have to match, and the reads would have to be sorted by path.

The three denials hold up whatever the reads were. The agent tried something the policy refused three times and still reported success ^[2].

The author does not argue that outcome grading is wrong. "Most coding benchmarks understandably care about the result," the post says ^[14]. The case is made with two hypothetical runs that both score PASS, one of which also tried the network, searched outside the workspace, attempted an unauthorized command and was blocked several times ^[13]. "Both may have produced the right answer. But they are obviously not the same run," the author wrote ^[16]. The evidence supports logging attempts beside the pass/fail column. It does not show the column should go, since in the author's own network example the sandbox held and the environment did not change ^[5].