Two Incommensurable AI Code-Review Benchmarks Each Rank Their Own Product First

An F1 score against the OpenSSF CVE Benchmark answers a narrow question: on code that contained a recorded vulnerability, did the reviewer flag it, and how often did it raise a false alarm. The corpus is public and holds more than 200 real production vulnerabilities ^[5], so a buyer can re-run the test. The number transfers if the failure you are paying to prevent is a security defect shipping in code shaped like that corpus. It does not measure whether the comment a developer received was readable ^[10].

LinearB's instrument asks the readable-comment question directly. It scored 16 bugs across two phases on competency, clarity, configurability and developer experience ^[1]. The result transfers if your review pain looks like those 16 bugs. DeepSource's corpus is roughly twelve times the size of that bug list ^[17]. LinearB's dataset was built in house, and the bug list and scoring criteria sit in a downloadable whitepaper ^[9], so the run is hard to reproduce against your own diffs.

The two write-ups disagree most sharply about CodeRabbit. In LinearB's telling it caught the most total issues but generated heavy noise, flagging the same pattern repeatedly without context ^[3]. In DeepSource's telling it scores 36.19% F1, 48.32 points below the 84.51% DeepSource reports for itself ^[6][16]. The dev.to write-up by Tessa Insley calls the two evaluations incommensurable, on the grounds that detecting recorded vulnerabilities and writing usable comments are different questions ^[10]. Greptile's self-reported 82% catch rate, DeepSource notes, comes from an internal benchmark of 50 pull requests across five repositories that has not been independently validated ^[7]. That averages ten pull requests per repository ^[19].

Both write-ups rank their own product first ^[8].

The statefulness finding is the one I would act on first, because it is cheap to verify: a two-week trial on your own repository tests it with one push. LinearB measured whether a reviewer withdraws or revises a comment once a later commit has made it stale ^[2]. Reviewers that withdraw outdated comments and revise after a fix scored higher on developer experience, and the tools that restart from zero on every commit pay for it in review time, because a fix that should close a thread instead opens a new one ^[12].

Configurability is the other axis both pages raise, and LinearB reports that YAML-defined rules and slash commands correlated with a smoother developer experience ^[13]. The configurability section of the write-up names Kodus among reviewers that put rules and signal ahead of raw finding count, without attributing a benchmark score to it ^[14]. LinearB also measures the average time from pull request open to the first correct, actionable comment ^[15]. Signal-to-noise, statefulness, configurability and time to first useful signal are all measurable on your own pull requests.