Skip to content

Build1 publisher3 min readPublished

Sorting the last 100 review comments finds 75% of the feedback a rule or a test can catch

The New Stack says AI-assisted teams have moved the bottleneck from writing code to verifying it, and the remedy it proposes is an afternoon spent sorting the last 100 review comments into rules, tests and judgment.

The Engineer · Build desk

Illustration accompanying Sorting the last 100 review comments finds 75% of the feedback a rule or a test can catch

What happened

  • A New Stack post on AI code review reports that teams with high AI adoption are merging 98% more pull requests, and that their review times have gone up 91%.
  • It cites one study in which 77% of engineers said they spend less time writing code now, with that time going into reviewing AI output.
  • The senior reviewers it describes hold 15 pull requests of 400 lines of code each in the queue every day.
  • The post rejects adding an LLM reviewer, on the grounds that a model that both writes and reviews the code shares its own blind spots.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint The 91% will not support capacity planning until someone says whether it clocks one pull request or a team's total hours, because the two readings point at opposite staffing decisions.
  • decision The 45% deterministic share converts review capacity into a tooling budget: someone has to write the checks and keep them green, and that person is not reviewing while they do it.
  • exposure Code that reads well and handles the happy path gets approved on sight, and the misaligned assumption becomes the approver's problem once it reaches production.
  • cost Preserving the intent behind a change means keeping prompts and agent sessions as retained artifacts, so whoever owns the pipeline inherits storage and plumbing that a diff alone never needed.

Multiply the two percentages in the post and the answer depends on what the second one measures [1]. If the 91% is the wall clock on a single pull request, the reviewing load is 1.98 times 1.91, or 3.78 times what it was [5]. If it is the team's total review hours, then time per pull request fell to 1.91/1.98 of its old value, about 3.5% less, and the whole increase is volume [6]. The post attributes its 77% figure to one study without naming it, and it does not define high AI adoption [22].

The per-reviewer number is concrete either way. Fifteen pull requests at 400 lines each is 6,000 lines of diff a day [4]. Some experienced engineers, the post says, point-blank refuse to review AI-generated code [23].

The author runs a community of senior engineers and engineering leaders, and says all of them name AI code review bottlenecks among their top concerns [18]. Volume is only part of the complaint. "When AI writes code, the reasoning is gone. The reviewer is left reverse-engineering intent from a diff," the author wrote [7].

The part that transfers is the sorting exercise, because you run it on your own history. Pull the last 100 review comments. Label each one deterministic if a rule can check it, execution-testable if running the code catches it, or genuine judgment [12]. The post reports a rough split of 45%, 30% and 25% when teams do this [13]. The first two buckets add to 75% [14]. "Three-quarters of review feedback is codifiable. Every recurring review comment is an invariant you haven't written yet," the author wrote [15]. The worked example is "New endpoints must have OTel spans", which the post calls an AST check and not a judgment call, written once [16].

Three of the five slop categories in the post look like rule and harness work to me. Convention-blind output ignores repo conventions around naming, error handling, logging patterns and module boundaries [20], which is the material an AST check encodes. Over-engineering shows up as a 200-line abstraction layer for a problem that needs 15 lines [21]. Hallucinated calls to APIs that do not exist are sometimes caught immediately and sometimes only in production, according to the post [25]. Plausible-but-wrong stays with a human: it reads coherently and handles the happy path, and catching it requires knowing what the code was supposed to do [10].

The objection to bolting on another model is that one which both writes and reviews shares its own blind spots, and that stacking adversarial agents turns engineers into "bot-sitters" who tune filters instead of building [11]. The recurrence test costs less to apply: if you have posted the same comment more than once, codify it [17].

What to watch

  • Whether anyone publishes the study behind the 98% and 91% figures with a population and a definition of high AI adoption.
  • Whether other teams run the 100-comment sort and publish splits that differ from 45/30/25.
  • Whether code hosts start attaching agent session transcripts to the pull request alongside the diff.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories