Build1 publisher3 min readPublished
A reviewerMode filter kept the agent's uniform replies out of its own batch audit
The rule required the longest reply in a batch to be at least twice the shortest. It never saw the 239, 269 and 291-character drafts a human queried, because the audit only read rows marked self-review.
The Engineer · Build desk
What happened
- On 2026-09-05 the shop owner looked at three replies the agent was about to send on Bluesky and asked why they were all about the same length.
- The three drafts ran 239, 269 and 291 characters, and each had already passed an independent review by a second model, one of them through six rounds.
- A batch-level audit added on 2026-08-15 would have failed that batch: its length rule required the longest reply to be at least twice the shortest.
- The audit sat behind a filter that kept only plan rows whose reviewerMode was self-review, and all three of these rows were graded by the subagent reviewer.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A property that only exists across a batch needs a gate that reads the whole batch. Adding review rounds per reply does not reach it, however good the rubric is.
- decision Adopting the 2x spread rule commits you to batches that include a genuinely short reply. Under a 300-character cap, one 291-character draft forces another of 145 characters or fewer.
- exposure The assertion logic was right the whole time and the defect was in the expression feeding it, so exercising the guard's logic would not have shown that production batches never reached it.
- cost For three weeks the owner's own reading was the only batch-uniformity check that ran. That is attention the guard was built to spend on his behalf.
The filter is one expression. `plans.flatMap` kept a plan only when `plan.toneReview?.reviewerMode === 'self-review'`, then handed the survivors to `assertSelfReviewBatchDiversityPassed` [10]. The mode is recorded on every plan row and the send CLI reads it [5]. Every row in the 2026-09-04 batch carried `reviewerMode: 'subagent'` [15]. So the array was empty, the audit had fewer than three replies to compare, and it reported no violations [11]. Technically a pass.
The premise for the filter went into the changelog the day it landed: a batch that had gone through independent review had already had a second pair of eyes on the same criteria [8]. The audit was hardened on severity that same day, from an stderr warning to a thrown error, after the owner pointed out that a warning the agent can read and ignore is not a gate [9]. The filter on its input stayed [10].
The premise fails for a structural reason. The reviewer sees one reply and one conversation at a time, and grades whether it is honest, in register, responsive and not condescending [17]. A 291-character reply passes all four on its own [18]. What was wrong lived at the batch level: three replies, drafted on 2026-09-04 for three different conversations [12], each written up toward the 300-character cap regardless of how much it had to say [19]. Six rounds of per-reply review cannot see a per-batch property [20]. The comment on the fix makes the point in one sentence: the reviewer looks at replies one at a time, so a habit of filling the batch to the cap is structurally outside its field of view [21].
The writeup gives the owner's question in translation: "the lengths are all about the same, is that intentional? There is no reason to make something long that has no reason to be long. Build a mechanism so each reply is as long as what it has to say." [16]
Price the rule before copying it. The lengths come from the approval ledger, which keeps the draft text: the 269-character draft was rejected, the 239 and 291 approved [13]. 291 over 239 is 1.22 against a threshold of 2 [14]. With a 300-character cap in force, a batch holding a 291-character reply clears the rule only if another reply in the same batch runs 145 characters or shorter [25]. For that gate to fit your pipeline, your batches need replies with genuinely different amounts to say, and they need at least three rows: below three the audit reports nothing [11]. Under the old wiring, a mixed batch with two self-review rows fell under the same floor [26].
The fix moved the length rule into its own function and put a wrapper, `assertReplyBatchLengthSpread`, in the shared CLI library [22]. All three senders call it, for public feedback, outbound engagement and direct messages, on every reply in the plan, with no filter on review mode, before the first network call [23]. Position matters as much as scope here, because the guard now reads the same list the sender is about to iterate. The writeup describes only the length rule being moved out; it does not say the opening-shape and closing-shape checks were rewired [22].
The audit landed on 2026-08-15 and the owner asked his question 21 days later [24]. In between, batches shipped with the check sitting in the codebase and unreachable from the path they took [15].
What to watch
- Whether the opening-shape and closing-shape rules get the same unfiltered wiring the length rule now has.
- What the team does the first time a legitimately uniform batch trips the thrown error in one of the three send CLIs.
- Whether the pipeline gains a check that the reviewerMode values appearing in production actually reach each guard.