Build1 publisher2 min readPublished
A two-parameter fit on your own merges puts a threshold under an LLM reviewer's "block"
A dev.to post argues that the approve and block labels AI review tools print are uncalibrated text, and proposes fitting sigmoid(alpha + beta * verdict) to a team's own review history so triage gets a threshold it can defend.
The Engineer · Build desk
What happened
- Every AI code review tool the author tested finishes by printing a hard label, approve, request changes, block, or a severity from 0 to 5, and that label becomes the thing the team argues about in PR comments.
- Asking the model for its confidence returns an uncalibrated number, because nothing in training taught it to map internal uncertainty onto the 1-100 scale it prints.
- The referenced post lists what a raw verdict cannot supply: trustworthy probabilities, threshold control to trade precision against recall, and any way to tell whether the model used the pasted context.
- The fix proposed is to feed the verdict as one input feature to a small model fitted on the team's own repository history and ground truth, then pick an operating threshold that matches how the team triages.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Before any threshold exists, someone has to label past findings against the team's merges and reverts, category by category. Those hours come out of review time.
- decision The cut between ignoring style findings and acting on security ones stops being a convention and becomes a number someone can remeasure.
- constraint A calibration fitted to one model version and one prompt has to be refitted on the vendor's release schedule, not the team's.
- exposure If the label is wired into a required status check, the merge gate sits at a threshold on an unestimated scale, and the false-positive rate is whatever the vendor's training distribution produced.
A code review tool is, functionally, a classifier: every finding comes back with a label and a severity. The LLM behind it was trained to produce plausible text, and an uncalibrated confidence score is plausible text too [15]. Ask a review model how sure it is that a finding is an auth bypass. "It will cheerfully say 0.98 whether it caught a real one or hallucinated a method call that doesn't exist," the post's author wrote [3].
The proposed replacement has one feature and two parameters: p(real problem in this file) = sigmoid(alpha + beta * LLM_verdict) [5]. Send beta to infinity and the model reduces to the raw verdict, which the post says is the behaviour teams already have [6]. A two-parameter model is not much of a model. That is part of the attraction: you can refit it whenever the tool changes.
The work sits upstream of the fit. The post's ground truth is your own merges and reverts. The fit then adapts to your codebase instead of to the distribution of code the model saw in training [9]. Someone has to pair each past finding with an outcome, and do it per category if you want per-category rates.
The post's illustration is a tool whose style findings are real 40 percent of the time and whose security findings are real 80 percent [8], with an operating point that actions anything above 0.8, logs the rest, and gets remeasured after two weeks [10]. It does not report measured rates from a named repository [14]. Take the figures anyway and the threshold sets the triage policy. With the verdict as the only feature and one label per category, the fitted probability for every style finding is that category's base rate, 0.40. That sits below the 0.8 cut, so the entire category lands in the log [13].
For a fitted beta to mean anything next month, the verdict distribution has to hold still. The author reports watching the same class of bug come back flagged "critical" on one run and "looks fine" on another run of the same model with a slightly different prompt [4]. Beta is fitted to one model version and one prompt string.
The reason to spend the labelling effort now is throughput. The survey the post cites describes the loop teams are converging on as generate, verify by execution, refine, with review and verification becoming the bottleneck as code generation gets cheaper [11]. The productivity work it cites reaches the same place from the other side: commits and lines of code can keep rising while end-to-end delivery stays constrained by review throughput and architecture fit [12].
What to watch
- Whether any review vendor ships a calibration curve or a per-repo fit alongside the severity field.
- Published per-category hit rates from a real repository, which would test the post's 40 and 80 percent illustration.
- A team reporting what happened to reviewer load after replacing the label with a fitted probability.