Security1 publisher2 min readPublished
One rule in CLAUDE.md lifted Huntress's API recall from 48% to 86% on the same seven tasks
Huntress ran seven Rails tasks past Claude Fable 5.1 and watched it copy hand-rolled code out of its own repository. Changing the harness, in three cheap steps, took API recall from 48 percent to 100.
The Watch · Security desk
What happened
- Huntress deleted shipped code from a pinned commit of its own Rails app and handed Claude Fable 5.1 the ticket that produced it, without naming the framework feature it expected the agent to use.
- With no guidance, the agent chose the built-in Rails feature in 10 of 21 trials, 48 percent, close to the 41.3 percent the Rails team measured publicly for the same model.
- That average covered two populations: features the model recognised, where it picked the built-in helper in 10 of 12 trials, and three features it never chose once in nine attempts.
- Adding one short rule to the project's CLAUDE.md file, with no other change, took measured recall to 86 percent.
- A lookup tool for the installed Ruby gems and a second model reading the diff before the agent finished closed the rest, reaching 100 percent across the seven tasks.
Compiled by The WatchSomething wrong?How this is made
Why it matters
- capability Recall is tunable by whoever owns the repository. A config file, a tool call and a review step moved it further than any model upgrade on the public board has.
- exposure has_secure_token sits in the zero-recall group, so a hand-rolled token generator already in a codebase is the shape agents will keep reproducing into new models and new pull requests.
- decision If one line in CLAUDE.md changes output across a whole suite, that file is configuration an engineering org has to review, version and own like the code it shapes.
- constraint The gain is measured in a single Rails application on a suite small enough to count by hand, so any other team needs its own evals before assuming its number moves the same way.
The transcripts explain the zeros. Huntress's agent greps the repository first, finds a hand-rolled version of the pattern it needs, and copies that [10]. One token generator sits in several of the app's models because one engineer wrote it and two AI-assisted pull requests copied it, and the eval benchmark copied it once more [11]. That is at least four copies of one generator [6].
Huntress checked each task against the installed gem source before accepting it, confirming the Rails feature fit and that both the Rails version and the hand-rolled version pass the hidden RSpec, so the grader scores the choice and not whether the code works [17]. Trials run in a Docker sandbox from a pinned commit with the databases up, no network beyond the LLM calls, and a turn and cost budget [13][14]. The hidden spec, the app's own specs and RuboCop on the touched files produce the pass [15]. A separate recall scorer reads the diff for the Rails shape against the hand-rolled shapes and checks the transcript for whether the agent ever identified the feature or opened the Rails source as a reference [16]. Each run changes one variable [21].
Twenty-one trials across seven tasks is three attempts each [1]. The 12 trials built on generates_token_for and normalizes cover four tasks; the nine built on perform_all_later, comparison: and has_secure_token cover three [8][9][2].
The public leaderboard gives the range between models: Sonnet 5 at 25.4 percent recall, Opus 5 at 34.9 percent [6]. Across the Rails benchmark's scores, from Sonnet 5 to the top figure for Claude Fable 5.1, that is 15.9 points of spread [5]. One short rule in a CLAUDE.md file moved Huntress's own number 38 points [3]. The two figures come from different codebases, so the comparison is indicative and not a controlled one. The gem lookup tool and the second-model diff review closed the last 14 points [4].
Huntress describes the failure mode as "handrolling what the programming language already gives us for free" [19]. In the has_secure_token case that means a callback plus a SecureRandom call where the framework helper is one declaration [20]. Because the agent copies what it finds, a hand-rolled generator already in the tree keeps propagating until something in the harness sends the agent to the gem source instead. The evidence base is seven tasks and a control in one Rails application, three trials each at baseline [18][1].
What to watch
- Whether Huntress publishes the wording of the CLAUDE.md rule and the gem-lookup tool so other teams can reproduce the 86%.
- Whether the Rails team's Agents on Rails benchmark starts reporting harness-configured runs alongside bare-model scores.
- Whether the second-model diff review holds up on tasks outside the seven, in repositories where the hand-rolled pattern is already widespread.