Loading today’s stories
Build1 distinct publisher3 min readUpdated
A thirty-run experiment finds agents followed long, buried rules anyway. The two failures came where the file contradicted the repository, not where it was long or buried.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A developer removed a 1,350-line CLAUDE.md from a working repository, ran the same ticket-sized task thirty times, and found that two rules out of twenty depended on the file existing [1][5]. That matters because the standard advice about agent rules files - that they are advisory rather than enforcement, that long ones get ignored, that anything buried deep gets skipped and belongs in a hook - is repeated everywhere and, according to the dev.to writeup by Mayank Kaul, has never been measured [3][4]. The design: twenty checkable rules, one task, six configurations (full file preloaded, file removed entirely, file on disk with auto-loading suppressed, nineteen lines plus a Skill, and two more arms on a stripped-down brief), with a deterministic checker reading the resulting file tree [5]. With the file present, 399 of 400 rule checks passed [6]. That works out to twenty runs with the file and ten without, and a single failed check across every arm that had it [1][2]. Seventeen of the twenty rules scored five out of five in every configuration, including the arms with no file at all [7]. Across six hundred checks only three rules failed anywhere, and two of them account for the entire gap [8]. Those two are not about length. Rule 12 says put a bullet under ## Unreleased; the repository contradicts it twice, since that heading has held the placeholder "- Nothing yet." for the project's whole history while all eight released sections carry the actual bullets, and the changelog's own header instructs you to cut a new version section and move the patch digit on every merged change [9]. Every run without the file read that header and followed it [10]. One reported moving to 0.4.3 "per the changelog's stated rule that the patch digit moves on every merged change" [11]. Rule 13 failed the same way: the file says bump the patch digit, the history shows 0.4.0 was a minor bump because it added new API surface, and three runs that added a public module picked 0.5.0, while a fourth left the version alone [12]. Kaul notes these are the only two rules in the set where the file contradicts what the repository demonstrates [13]. The burial claim does not survive the layout. Rules eleven through twenty all sit below line 314, under a heading in which the file itself says these are the ones most often skipped in practice [14]. Seven of those ten never failed anywhere, and rules 12 and 13 failed only in the arms without the file, which cannot be a burial effect because burial requires the file to exist [15]. The one remaining buried-rule failure did occur in a run that had the file, though the supplied material is cut off before the detail [23]. The instrumentation notes are the part worth stealing. The first version of the experiment found no effect at all, because deleting the file left it in git and git status printed D CLAUDE.md in the first orienting command; ten of ten runs recovered it and six said outright they had read the contract out of HEAD [16]. Two further channels leaked: a memory plugin installed months earlier and forgotten, and Claude Code's own per-project memory directory [17]. The base repository was rebuilt so the file was never committed at any depth, verified by scanning every blob in the object store [18]. Kaul names the residual confound himself: the test harness carries a comment pointing at CLAUDE.md, four of the ten no-file runs reported the file missing, and one said it followed the conventions the committed modules demonstrate instead - so those arms are a model told a contract exists and unable to reach it, not a naive one [19][20][22].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The experiment used twenty checkable rules in a 1,350-line CLAUDE.md, one ticket-sized task, and thirty runs across six configurations: the full file preloaded, the file removed entirely, the file on disk with auto-loading suppressed, nineteen lines plus a Skill, and two more on a stripped-down brief. A deterministic checker reads the resulting file tree.
With the file present, 399 of 400 rule checks passed.
Rules eleven to twenty all sit below line 314, under a heading where the file says of them, in its own words, that in practice they are the ones most often skipped.
Seven of those ten deeply placed rules never failed anywhere, and rules 12 and 13 failed only where the file was absent, which cannot be a burial effect because burial only exists when the file exists.
One buried-rule failure occurred in a run that had the file present; the supplied source text is truncated before the detail is given.
A developer deleted a 1,350-line CLAUDE.md from a project and reported that two rules broke.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One documented self-run experiment, unreplicated
The design is unusually explicit for a practitioner post: six hundred deterministic checks over twenty rules, six configurations, an ablation of the brief's 'follow the existing utilities' clause, and a disclosed correction after three context-leakage channels invalidated the first attempt, verified by scanning every blob in the object store. Against that: a single publisher and single author, no independent replication, no linked artifacts in the supplied text, one repository and one ticket-sized task, per-arm counts as small as three or four runs, an author-named confound in the no-file arms, and a source body that truncates mid-sentence.
Single-harness benchmark, no external uptake
The only observable uptake is the author's own harness: one thirty-run benchmark plus a per-run cost figure for the Skill configuration, both from the same setup on the same day. No third party is reported running the checker, reproducing the numbers, or changing a rules file in response, and the supplied source names no organizational deployment. The disclosure of hidden context channels in Claude Code and a memory plugin evidences tool usage but not adoption of this finding.
Mostly calibrated, generalization runs ahead of n
The framing largely deflates hype rather than adding it: the headline number matches the reported checks, the author discloses that his first attempt reached the opposite conclusion, names his own confound, and runs the obvious counter-ablation. The overstatement is in scope, not arithmetic — conclusions such as depth explaining nothing and rules being inert unless the codebase contradicts them are drawn from one repository, one task and per-arm counts as low as three or four runs, with no model or version stated and no replication.
Independent author, engagement upside on a contrarian result
The source is a self-published developer post on dev.to with no vendor, sponsor or employer relationship disclosed in the supplied text, so there is no evident commercial stake in the outcome. The residual incentive is attention: a counterintuitive headline about deleting a large rules file rewards a strong claim, and the author is the sole builder, runner and scorer of the harness with no external check. Offsetting that, he reports discarding two drafts built on a result he later found wrong and publishes the confound and the ablation that weaken his own story.
Internally coherent, externally unverified
The numbers reconcile across the piece — six hundred checks over twenty rules implies thirty runs, four hundred file-present checks implies twenty runs with the file and ten without, matching the 'ten of ten' and 'four of the ten' counts — and the method is described specifically enough to challenge. Confidence is capped by the single-source cluster, the absence of replication or linked artifacts, the unnamed model, small per-arm counts, the author-named control-arm confound, and truncation of the supplied text.
build
AI-written code fails the same four ways, and every gate you own reports green1 distinct publisher
build
2,513 tool calls, zero refactorings: what agents actually do when you ask them to refactor1 distinct publisher
build
The failure modes of AI agents in low-level code have names, and names can be gated in CI1 distinct publisher
build
Block's Berd makes a duller argument than its mascots: show the agent's context as product state1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 16, 2026