CauterRule traced its 0.08 extraction score to a token-F1 gate inside the metric

Token F1 over a directive is a set comparison, and the directives are one line long. "pull latest changes before pushing" and the accepted rewording "run a rebase or fetch-merge first" share almost no tokens, and the pair scores about 0.22 ^[8]. The first version of the metric gated its agreement number on exactly that score ^[7]. A composite gated on its most brittle component cannot rise above what that component allows, so on the golden corpus the headline agreement fell to 0.08 to 0.10 ^[9]. The trigger, the part of the rule that names the failure, was matching at semantic similarity of 0.6 or better in the same runs ^[10]. The trigger sub-metric ran six to seven times higher than the gated headline ^[24].

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures, replay-tests them, and tries to separate reusable guidance from overgeneralization ^[2]; v0.3.1 is on GitHub and PyPI ^[1]. The two changes here sit in the reporting layer. agreement is now the trigger-only semantic match, with the directive score published alongside as directive_f1 and no longer deciding the headline ^[11]. Every sub-metric is printed next to the composite ^[12]. The models and thresholds stay as they were ^[21].

The project had already accused its own replay gate of comparing text lexically, then shipped a new metric that did the same thing ^[14]; the write-up calls the first version "a lexical comparator wearing a validation costume" ^[13]. The metric answers one question, whether the model extracted the right rule. Replay answers the other, whether the rule can be verified against history ^[26].

Two smaller fixes change labels. extraction_f1 was emitting 0.0 for corpora that carry no expected_rule at all, including raw/ci; to_dict() now returns null when n == 0 and the runner emits None ^[17]. On raw/opencode, rates of 0.68 and 0.72 across the two models had been reported as false_accept_rate when the corpus is a promotion corpus and the figure is its acceptance rate ^[18]. The wrong name, the write-up says, makes "a pass rate read like a safety violation in every summary file we shipped" ^[20]. Rejection corpora such as nearmiss and adversarial/* keep false_accept_rate; silence and extraction corpora now report acceptance_rate ^[19].

For the 0.6 trigger figure to mean anything on another corpus, the ground truth has to be worded as variously as the model words its output. Four of the 40 corpora carry an expected_rule field ^[6], one in ten ^[22], covering 386 trajectories once the 15-trajectory paraphrase-diversity slice is counted inside reference-expansion's 303 ^[23]. Golden's 60 entries were backfilled with authored scenarios in issue #735 ^[6], so the expected wording came from the same project that built the extractor. The write-up does not name the model behind the semantic comparison ^[25]. A trigger-only gate inherits whatever that model calls similar. The field test covers two cloud models and 4,742 trajectory-runs ^[3].

The new behaviour is pinned by test_agreement_is_trigger_only_directive_not_gated in tests/measurement/test_extraction_accuracy.py ^[15]. The write-up says that a metric which gates a release is code, and it gets tests like anything else ^[16].