A review comment becomes a gate only when someone can write an assertion for it. The authors mined merged pull requests, kept the comments that asked for a change rather than the ones that mused, and turned each surviving rule into a constraint test [2]. Their published examples show the spread: "Please add type hints to all function signatures" is a checker rule, while "Move this logic into a separate method for testability" is a judgement that somebody had to encode as structure [8]. The write-up gives aggregate figures and example rules; it does not publish the constraint test bodies [20], so how tightly that second kind is scored is not visible from outside.
The staging is the part worth copying. Stage two runs only if stage one passes, and the reason given is to keep "the agent can't solve the problem" apart from "the agent solves it in a non-mergeable way" [9]. That makes the 221 a count of correct patches that turned out unmergeable, separate from a general failure count [3]. It also produces a paired score, two figures instead of one [14]. That resistance to a single column makes the benchmark harder to put on a slide, and that awkwardness may be its most useful property.
Four backends across 303 instances is 1212 attempts if every backend ran every instance, which would put the functional passes at roughly 53% [16]. The post does not state the attempt count [20]; that 53% figure is arithmetic built on an assumed total, not a reported result. It also names GPT-4 and Claude without versions and leaves the two open models unidentified [21], so the headline rate cannot be pinned to a model generation.
For 34% [15] to be your number, two conditions have to hold. Your review rules have to be as mechanisable as the ones mined from 75 Python repositories [4][2]. And your pipeline has to be doing what the post assumes when it says these patches would merge in a test-only CI pipeline [10], which is running tests and nothing else. The largest published failure category is missing type hints, at 18% of constraint failures [6], or about 40 of the 221 patches [19]. If your CI already runs a type checker, that slice never reaches a reviewer, and the post publishes no shares for the remaining categories [20].
The adoption cost sits where the source puts the constraints: review guidelines, team norms and the implicit knowledge of senior engineers [12]. Nothing in the benchmark hands you your own repository's rules, so you author those tests. Execution is the cheap half. The harness caps each pytest invocation at 300 seconds and shells out through subprocess.run in list form [11], so a patch that reaches stage two can burn 600 seconds of wall clock [17], and 1212 patches at that ceiling is about 202 hours run serially [18].
