Build1 publisher2 min readPublished
CauterRule's Domain-Scoped Replay Lifts Golden Recall from 0.087 to 0.17-0.228, Still Short of 0.70 Goal
Both field test reports pointed at replay and matcher calibration, but v0.3.0 fixed the recall denominator with one list comprehension that scopes each candidate's references to its own domain, and the reported number roughly doubled.
The Engineer · Build desk
What happened
- Golden recall stayed at 0.087 across two CauterRule field tests, and both reports named replay and matcher calibration as the highest-value engineering target.
- Recall divided a candidate's catches by the whole reference corpus, so the metric was asking whether a git rule also prevents docker, python, terraform and browser failures.
- Version 0.3.0 scopes each candidate's reference set to its own domain before replay, and the overall reference pool grew from 230 to 444 trajectories.
- The same release cut the pass threshold from 0.8 to 0.5, and golden pass rates came in at 40 percent for gpt-4o-mini and 50 percent for llama-3.1-8b.
- The v0.3.0 field test covered two cloud models, 40 corpora and 4,768 trajectory runs, with post-fix recall between 0.170 and 0.228.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Anyone reading a recall number out of a replay harness now has to decide what population each candidate was judged against, and that decision requires a domain label on every reference trajectory.
- constraint A wrong denominator does not throw an error; according to the post it depressed every candidate equally, so the team read a capability problem where an evaluation problem existed.
- contradiction Because the scoping change and the threshold drop shipped in the same release, the pass-rate figures in the v0.3.0 report cannot size either change on its own.
- exposure Rules learned in a domain with few failing references are now graded against a small pool, and the post says it has not measured whether that penalizes them.
The filter is one list comprehension: `references = [t for t in reference_pool if t.domain == candidate.source_domain]` [6]. Nothing downstream of it changed. The post says there was no model change, no prompt change, and no change to the core matcher scoring [10]. For that line to run at all, every reference trajectory needs a domain field and every candidate needs a source domain. Anyone copying the fix has to do that labelling first.
Scoping took the worked example from 3/200 = 0.015 to 3/~27, about 0.11 [5][8], a factor of roughly seven [1]. The pass threshold went from 0.8 to 0.5, a factor of 1.6 [9][2]. The post attributes the 2-3x recall gain to scoping and the pass-rate movement to both changes [14], which is more than most benchmark tables bother to separate.
The per-domain figures in the post do not reconcile. The worked example divides three catches by about 27 git failures [8], while the limitations list says git has 19 references against roughly 30 each for python and docker [13]. Failing references are a subset of references, so a 19-trajectory git slice cannot produce a denominator of 27 [6]. Either the ~27 is illustrative or the domain counts come from a different snapshot.
For the doubling to transfer to another harness, two things have to hold. The reference trajectories need a domain label that partitions failures the same way the rules do. And each domain needs enough failing references that a single catch is not most of the score. At a denominator of 27, one prevented failure is worth 0.037 of recall [4]. At a denominator of five, one catch is 0.2 and the promotion decision is close to a coin flip.
The post is direct about what scoping did not fix. Semantic matching runs at a 0.2 blend weight and cannot connect "non-fast-forward" to "Updates were rejected because the remote contains work that you do not have locally" [12]. Both strings describe a rejected push, and the matcher sees different tokens.
Post-fix recall of 0.170 to 0.228 sits against a stated ambition of about 0.70 [11], so the top of the range is roughly a third of target [5]. The reason the write-up gives for believing the problem was measurement is that the improvement held across both cloud models, which it calls "the signature of an evaluation bug, not a capability ceiling" [17]. In any agent harness where every candidate scores badly at once, check the shared denominator first: it drags every candidate down by the same amount, and the earlier reports took that uniform failure for a matcher problem [16].
What to watch
- Whether CauterRule publishes per-domain recall showing if thin domains like git are penalized against reference-rich python and docker.
- A rerun that changes scoping alone at the old 0.8 threshold, which would size the two changes separately.
- Whether raising the 0.2 semantic blend weight closes the paraphrase gap without costing precision.