A delta needs a baseline. "+3 complexity" says nothing on its own, so each model patch is scored against the human commit that closed the same issue: check out the base commit, measure the files the patch touches, apply it, measure again, subtract [5]. The human diff runs through the identical pipeline and trips the same regression rule on 24% of instances [8]. That is the number worth holding onto. It says the bar is low enough that ordinary human bug fixes clear it about one time in four, and the models still fall the other side of it more often [2].
Only two of pyscn's metrics are in play, and the reason is partial-repo analysis rather than convenience. Cyclomatic complexity and dead code both fall out of the control-flow graph of one function in one file, so they stay exact when you analyze a subset; coupling, cohesion and clone detection undercount whatever the unread files would have contributed, which would make the score depend on which files the patch happened to open [6]. Two housekeeping rules follow from the same problem. Files present on only one side are excluded, because a file with no before-state inflates every delta it lands in, and a deletion books a whole file's complexity as an improvement [9]. Scoring is restricted to patches that passed their tests, which cuts against the finding rather than for it: ungated, the sample roughly doubles and eleven models clear Bonferroni instead of six [10].
The headline count is a sign test. Thirteen worse, one tied, none better is p = 0.00024 against a coin flip [11]. Per model it thins out: twelve of fourteen clear p < 0.05 alone, six survive Bonferroni across the fourteen comparisons [12], so six are significant only until you correct for having looked fourteen times [4]. The disagreements are where the effect is legible. On 741 issues exactly one side regressed, and the model was that side on 530 [14], leaving 211 the other way [1], about 2.5 to 1 [3].
The file litter is a different mechanism from the complexity. An agent writes throwaway scripts as it works, reproduce_bug.py, check_url_parts.py, final_verification.py, and the final patch is a git diff, so anything still sitting in the working tree ships with the fix [16]. A file named final_verification.py arriving on main has not verified much. Human patches create 0.00 files the model's does not, while models reach 5.00 [17]. Under a second framework, gpt-5 leaves 0.07 and claude-opus-4.5 leaves 0.00, against kimi-k2 at 3.84 [19], roughly fifty-five times gpt-5's rate in the same harness [5]. Those files cannot be what produced the complexity result: a new file has no before-state, so the exclusion rule drops it from the delta [6].
What does not transfer is the ranking. The biggest gap belongs to devstral-small, near the bottom of the leaderboard, with gpt-5 second, both on OpenHands; across the seven models sharing an identical framework there is no trend, with o3 and o4-mini above claude-sonnet-4 and qwen3-coder-480b [15]. Django alone gave three worse and two surviving correction, and the other eleven repositories mostly bought statistical power [13]. For this to carry to your repo you need Python, acceptance that cyclomatic complexity and pyscn's high-risk rating stand in for health [3], work shaped like closing an issue on an existing project [1], and a harness that emits the working tree as a diff [16]. The patches are public leaderboard submissions the authors downloaded rather than ran [4], so the hygiene being measured is partly the submitter's harness settings.
The gate this evidence supports is narrow and has two checks: files absent from the base commit get rejected unless the diff explains them, and the complexity delta on the edited functions gets read against what the fix actually required. Every result set is committed, so both halves can be rechecked against your own thresholds [20].
