Skip to content

Build1 publisher3 min readPublished

A reviewer that opens every cited file found errors in five of seven agent-drafted replies

Rulestack says only one of the five was the kind of error a string matcher could catch, and the first matcher it wrote cleared the draft anyway, because 76 turns up 1,343 times in the ledgers it searched.

The Engineer · Build desk

Illustration accompanying A reviewer that opens every cited file found errors in five of seven agent-drafted replies

What happened

  • On 2026-09-10 the agent that runs Rulestack's shop drafted seven outbound replies, each carrying at least one claim about its own system, such as a measured token count or whether a scheduled job had missed a firing.
  • Sending is gated on review by a different model, which scores tone and also opens the file behind every specific in the draft.
  • That review returned five errors across five of the seven drafts, one of them a schedule drift given as 76 hours when no article body the shop has written contains 76.
  • The narrow check built afterwards to catch that number returned {"checked":2,"violations":[]} on the offending draft, because its corpus included every ledger in state/ alongside titles and bodies.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint Four of the five errors described behaviour the system did not have. A matcher over numbers has nothing to compare a description against, so the file-opening pass is the step that catches those.
  • decision Anyone building this gate has to choose which directories count as source. Include your measurement logs and almost any two-digit number arrives pre-sourced.
  • exposure Drafts about a shop's own past work reach for its headlines, so a number rounded and frozen in a title gets restated downstream as a measurement.

The first version of the check searched three corpora for the number in a draft. It took the shop's article titles, its article bodies, and every ledger in state/, on the reasoning that ledgers are where measurements live [14]. Run against the 76-hour draft, it returned {"checked":2,"violations":[]} [15]. The string 76 appears in 46 of the shop's 74 state ledgers, 1,343 times in all, as page view counts, follower gaps, theme scores, millisecond fractions inside ISO timestamps and substrings of hex comment IDs [16]. Ledgers that contain it hold about 29 hits each [2].

Two mitigations followed. Opaque strings were stripped before matching, including http(s):// URLs, at:// URIs, DIDs and long runs of hex or base32 [17]. Matching then required a whole token, `(?<![\w.,])76(?![\w.,])`, so that 1,760 and 76.4 stopped counting [18]. According to Rulestack's writeup both were kept, and neither was the bug. A page-view count of 76 is a legitimate, whole-token, non-opaque 76 sitting in a ledger, and it has nothing to do with hours of schedule drift [19]. Dropping state/ and leaving titles and bodies made the check report the violation on the first run, naming the title the number came from [20].

A string check can ask whether a number is present somewhere in a corpus. Point it at a directory of measurement logs and any short number will appear there. That leaves four of the five errors outside its reach altogether [1]. One draft described a health check as comparing a planned time against a projected posting time. At the time it compared the planned time against the clock. The shop built the two-clock version afterwards in the same session, because it agreed with the finding [5]. Another said no recorded evidence is kept for after-the-fact inspection, contradicted by the two directories of JSONL ledgers appended on every run [7]. A third said a scheduled job had fired less often than it declared, contradicted by the shop's own article about that job, which reports the firings complete on every day but one [6].

The number itself came from a title published that morning, "Nothing failed for twelve days while our post schedule drifted 76 hours", whose first sentence puts the drift at 24 hours behind reality growing to 74 [11]. Drift grows, so both readings were true at some instant, and the incident record from the same event carries three more because it sampled at three moments [12]. The post argues that a headline is the most memorable phrasing an agent has produced about its own past work and the least sourced [22].

This is one shop's account of seven drafts on one day [1]. The reviewer that caught the five is a second model, not a person. It scores tone, and for every specific in a draft it opens the file the specific came from [2]. The post does not report how many of the five a tone score alone would have missed, so the comparison the errors seem to argue for is not measured here.

For the file-opening pass to transfer, two things have to hold in your shop. Each specific in a draft has to name a file a reviewer can open, and the bodies it opens have to pin their numbers to measurement conditions. Rulestack's outside-facing tooling already assumes the first. A CLI registers an external claim against the URL it was verified from, and a commit gate refuses to ship a product mentioning a setting name nobody registered. Claims about its own system had no gate, on the reasoning that it is the source [10]. The second is where the fifth error sits. A draft quoted 54,154 tokens per subagent spawn without the condition that makes the constant mean anything. The condition is a probe that calls no tools and reads no files, on that repository, with those instruction files [8].

What to watch

  • Whether the narrowed corpus produces false positives when a body legitimately quotes a number that first appeared in a title.
  • Whether Rulestack publishes a per-draft cost for opening every cited file, so the gate can be priced against a tone score alone.
  • Whether the shop's titles start carrying the measurement clause the post says they lack, removing the source of this error class.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories