The verdict that dominates this field test comes out of a string comparison. Replay takes a candidate rule, fires its trigger at reference trajectories, and scores two things: whether the trigger hits the cases it should (recall) and stays quiet on the ones it should not (precision) [6]. High enough on both and the candidate passes; clearly firing on wrong cases or missing every right one and it fails; anything in between comes back inconclusive [6]. The matcher doing that deciding is substring and token overlap, checking whether the words in the trigger appear in the trajectory text [7]. So the gray zone is a band of overlap scores, and its width is a tuning parameter of the tool.
That changes how the per-model rates read. The report's own explanation for the 42.9% floor is that better extraction produces more specific triggers, which are easier to verify [8]. Restated in matcher terms: models that emit distinctive tokens land outside the band more often. "when git push fails with non-fast-forward" gives the matcher something to hit, and "when a command fails" does not [13]. The 20-point spread between that floor and gpt-4o-mini's rate [17] is therefore measuring how lexically specific each model's trigger prose is, which is at least as much a property of the extraction template as of the model. The article gets to a similar place by calling the output less scorable rather than worse [15].
Arithmetic on the part that did resolve. The three pass counts given are 77, 72 and 93, and the article does not say which model owns which [10]. They total 242, which is 15.7% of the 1,538 candidates, and that is three of the four models [16]. Across the 394 trajectories, the run produced about 3.9 candidates each [18].
For those rates to transfer, your trigger text and your trajectory logs would need to share vocabulary the way this corpus does. Structured logs that keep error strings verbatim, or an extraction prompt that quotes the failing command, both narrow the band. Change the prompt and the ranking moves.
The report is candid that the undecided bucket holds two mixed populations and that the data cannot separate them [13][14], and it names replay and matcher calibration as "the highest-value engineering target" [11]. The cheap instrumentation is per candidate: log the overlap score the matcher computed, plus the token count of the trigger. Sorting the gray zone by those two columns splits vague triggers from matcher misses without buying a model judge.
Adoption is a pip install, with a CLI, an MCP server, seven export formats and a bundled git rule pack [2]. What to weigh is that promotion to a standing rule is gated on a verdict the project itself lists as its top calibration target, alongside an admission that the matcher and simulator can over- or under-fire on substring overlap [11][12]. Every figure here comes from that one self-published field test [3], and the counts behind "more than half" do not appear in the text [19], which leaves the majority bucket as a stated proportion rather than a number anyone else can recompute.
