An integration test earned its keep because someone wrote the expected intermediate state down before the code ran. Agents break that arrangement by acting. There are tool calls, multi-step reasoning, and failures that compound inside one run, where a conventional LLM eval only ever had a single output to mark [2]. Grade the output alone and you lose the why, which is the part you need in order to change anything [5]. The configuration list makes that concrete: if the baseline is a system prompt plus `read_file`, `edit_file` and `run_tests` [12], then any failure has to be attributed to the prompt or to the tools, and a pass/fail on the final patch cannot tell you which one moved.
The presence of `run_tests` in the toolset is the quiet load-bearing choice. A test suite is a deterministic checker, which is why bug fixing is the easy case for this kind of harness. Where a task has no equivalent, a trajectory score has nothing to sit on, and what is left is a semantic judge, the move the field made once surface-overlap metrics like BLEU and ROUGE stopped carrying information [13]. That does not remove the reliability question, it relocates it onto the judge.
The two suites the author separates should also never be averaged. Capability evals aim at unsolved tasks and expect a low pass rate, because the target is finding the ceiling [6]. Regression evals aim at solved ones and expect near 100%, because the target is catching breakage when a prompt is edited or the model underneath is swapped [7]. A single combined score moves for opposite reasons and reports on neither.
The arithmetic is worth doing because it removes an excuse. Three configurations over the same ten cases [11][1] is thirty documented task runs, which puts a documented run somewhere around three cents [1]. Nothing here is gated on inference budget. What is gated is labour of the unglamorous kind: each case needs a written root cause and a verification step, and the trajectory log has to record which tool was called with which parameters at which step, or the failure attribution the whole approach promises does not actually exist. That is the difference between an eval harness and a demo that passed once. The author is explicit that this was a personal experiment written up rather than a validated method [15], and the honest read of it is a shape worth copying rather than a result worth citing.
