Build1 publisher3 min readPublished
Fixing 62 true review comments left AgentCoop guarding a config key nobody writes
On PR #149, Codex filed 68 comments across 10 rounds and the team fixed 62 of them, all correct. The write-up blames the house rule that turned every comment into a commit for the code that came out worse.
The Engineer · Build desk
What happened
- PR #149 on the AgentCoop project drew 68 Codex review comments across 10 rounds, the team fixed 62 of them, and the write-up says every one of the 68 pointed at a real problem.
- The project requires every Codex comment to be fixed or refused in writing before merge, and the author says dealing with a comment quietly became fixing it every time.
- On PR #159 a new strict config check meant an upgrade could stop a healthy service and then refuse to restart it, and Codex marked that consequence P1.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Each accepted fix buys protection against a rare or hypothetical failure at the price of code every later reader must understand and keep working, and the people who pay that price were not in the review.
- constraint When the gate accepts either a fix or a written refusal, the fix is the cheaper exit, so a highly accurate reviewer pushes a team toward changing code it should have left alone.
- decision A severity label only means something against a promise somebody made, so a team running automated review has to write down what it guarantees before it can triage a P1 honestly.
- precedent Because comments arrive with a file name and a line number, that line becomes the default place to change, and the tool's output format ends up choosing where the codebase moves.
A fix is a new surface, and the next round of review inspects it. One function on PR #149 went from 28 lines to 42 across six consecutive rounds, each round finding a small gap that the previous round's fix had opened [2]. Fourteen lines added, a 50% increase, about two and a half lines per round [2]. The outermost layer keeps change-detection from confusing a config key written as the number 1 with the same key written as the text "int:1" in the same place [3]. It is possible to write that config file that way, and nobody has [3]. The author wrote that the protection "now lives in the code forever, for everyone who touches that file to read, understand, and keep working" [4].
The house rule on AgentCoop is that every Codex comment is fixed, or turned down with a written reason, before the pull request merges [8]. The two paths cost different amounts: a fix is a commit, and a refusal is an argument with a reviewer that had been right on all 68 comments [1]. Six were turned down and 62 were fixed, about 91% [1]. The author wrote that "we had a routine for fixing a comment and no routine for deciding about one" [15].
Scope creep runs on the same ratchet. A change on PR #116 read saved records and nothing else, until the reviewer noticed the records could be out of date, so the change grew the ability to write them [10]. Writing changed how saving worked, saving needed a new piece of running state, and the new state changed how a restart picks up where it left off [10]. Four rounds, every step a real point correctly fixed, and the team's eventual move was to delete all of it [10].
PR #159 added a check to `coop start`: read the config file properly, refuse to start if it is broken [16]. The upgrade command stops the service, downloads the new version, then starts it. So an upgrade could stop a healthy service and then refuse to bring it back, if the config passed the old version's rules and failed the new one's [17]. Codex marked it P1, top priority [17]. The human co-owner of the project, called 老哥 in the post, asked which piece of open-source software promises that an upgrade will never interrupt a running service [18]. He also pointed out that the guard Codex wanted would run before the upgrade, under the old version's rules, against a problem only the new version's rules can see. It would shrink the window and not close it [19].
For these numbers to say anything about another codebase, three conditions have to hold: comments arrive with a file name and a line number [13], the reviewer is accurate enough that arguing with it feels unreasonable [1], and the merge gate demands a disposition on every comment [8]. AgentCoop is a gateway that puts AI agents into Rocket.Chat and Mattermost rooms to talk with people and with each other [6]. The review load here is self-reported by one of the agents that writes the code, working with the human co-owner [7]. The post does not give the total lines the 62 fixes added, or a comparison against human review of the same repository.
The author wrote that "None of this is Codex's fault. It reports what it sees, and what it sees is real" [14]. In my view the gate is worth keeping. The refusal form is the part that needs work: how often the case occurs, and who reads the line afterwards, written next to the proposed fix. By the post's own account the two sides were never put on the same scale [12].
What to watch
- Whether the post's promised Chapter 1 gives a usable test for turning a true comment down, and whether it survives a comment labelled security.
- Whether Codex starts attaching a frequency or cost estimate to a comment, which would move the trade-off out of the queue and into the tool.
- Whether the team publishes line counts before and after the 62 fixes, so the permanent maintenance cost can be measured rather than argued.