A generic loop-cost harness for any OpenAI-compatible endpoint asks the model once per repeat

apply_patch is the part I would keep. It scans the reply for fenced diff or patch blocks, takes the last one, and pipes it into `git apply --whitespace=nowarn` ^[13]. No block at all returns -1, which the code comments as the model answering with prose instead of a patch ^[13]. A block that will not apply returns -2 ^[13]. Both codes land in the row beside the test result, so a pass that produced nothing to test is still a row ^[14]. The post says its records include failures, "which are usually the interesting rows" ^[3].

Each repeat calls ask() once, and ask() sends a single user message at temperature 0 ^[11]. The printed invocation uses `--repeats 5` ^[8]. Five repeats at one request each is five chat completions and five rows for one task, none of them containing a second turn ^[17]. Variance across identical prompts is worth having, and the post is straight about why it is not zero: temperature 0 "reduces one source of noise, though it does not eliminate provider-side nondeterminism" ^[15].

Before each repeat the script forces a checkout of the pinned commit and runs `git clean -fdx` ^[10]. The -x takes ignored files too, so a virtualenv or a node_modules inside the scratch copy goes with them ^[19]. For the example task, `pytest -q tests/test_parser.py`, that costs almost nothing ^[8]. For a repo that needs an install before its tests run, every repeat pays that install, and the script installs nothing ^[10]. The seconds field is measured with time.monotonic around the HTTP call ^[12], so setup and test wall clock never reaches the file ^[20].

usage arrives as `payload.get("usage", {}) or {}` ^[12]. An endpoint that returns no usage block leaves prompt_tokens and the completion columns empty, and the client-side seconds is then the only cost number in the row ^[21]. So before you compare two tiers with this CSV, send one request to each and confirm usage comes back. The export block leaves LLM_MODEL as a placeholder, because the right value depends on what the provider currently serves ^[7]. The urlopen timeout is 600 seconds ^[11]. That puts a ceiling of 3,000 seconds, or 50 minutes, of request time on one five-repeat task against one endpoint ^[18].

The opening anecdote is the stronger version of the cost argument: a reviewer sent back a pull request carrying three different fixes for the same flaky parser, all green in CI, and the repo paid the difference in review time because the agent was free to run and nobody counted ^[16]. The row records the run number, request seconds, token counts, the diff shape and the test exit code ^[14]. It has no column for a reviewer's hour. If I were adopting this, I would keep apply_patch and its two failure codes, wrap ask() in a turn loop that feeds test output back into the prompt, and write the test wall clock into the row next to the request seconds. The disclosure that the article was prepared as part of MonkeyCode's product outreach is printed in the article itself ^[5]. That is more than most outreach manages.