Build1 publisher3 min readPublished
Hand adjudication cleared every swallowed-error flag in 120 local model generations
A naive Semgrep rule fired four times across 120 generations from a 1.5B coder model and none of the four survived review, because the rule inspected try/except while the suspect default returns sat behind if guards.
The Engineer · Build desk
What happened
- Qwen2.5-Coder 1.5B ran locally on CPU through Ollama across 12 frozen tasks at 10 seeds each, temperature 0.7, producing 120 analysed samples.
- A naive Semgrep detector flagged four swallowed-error candidates and hand adjudication found zero true positives: two came from an appended usage example, two were documented fallbacks.
- TypeScript went the other way, with 33 of 50 failure-path generations writing try/catch that logged, re-threw, or both.
- The author began the project convinced small models routinely swallow failures and set out to measure a contamination rate; the measurement contradicted that starting assumption.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A gate scoped to try/except passes a file whose only questionable line is a bare default return behind an if guard, so a green build covers try/except handlers and nothing else.
- decision Teams adding a swallow rule to CI have to staff the adjudication step, because telling a documented contract from a swallowed error needs the function's role and the caller's expectations, and the file contains neither.
- contradiction AIRA's 1.80x gets quoted as proof that AI code swallows errors while the paper itself says a flag is not a defect, so the argument between the two results is about who issues the verdict.
- exposure Citing this work as evidence that small models rarely swallow errors means generalising from 12 tasks on one model family, labelled by a single rater.
Semgrep fired on what it was written to match: try/except blocks whose handler absorbs the exception [5][6]. Four hits in 120 generations, and review cleared all four [5]. Two sat outside the function under test, because the model had appended a usage example with an empty catch after the function, and the rule read the file [5]. The other two were fallbacks the generated code documented in a docstring or comment as its contract [5].
The cases worth arguing about were if-guard default returns, and the rule matched only try/except [6]. Of the 50 Python failure-path generations, 39 used no try/except at all: 14 raised through an if/else, 21 let exceptions propagate, and 4 returned None behind an if guard [7]. Eleven, then, used try/except [18]. Those 4 if-guard returns are 8 percent of the Python failure-path sample [19].
Reading them takes context. In parse_int, returning None when the input cannot be converted is documented as the contract [10]. In fetch_json, every non-200 response collapses into the same None while a network error raises instead [10]. That leaves the caller three behaviours to track, and the generated code carries a comment saying so [10]. The author's conclusion is that the deciding information, meaning the function's role, the caller's expectations and the spec, lives outside the pattern, so static rules can surface candidates but not deliver the verdict [9].
The counterweight cited in the write-up is AIRA, a 2026 preprint (arXiv:2604.17587) that ran a deterministic, parser-backed analyzer over 955 AI-authored and 955 human-authored samples and reported 1.80x more high-severity findings in the AI set, 0.435 against 0.242 per sample [14]. The difference is 0.193 findings per sample, about one extra high-severity finding for every five AI-authored samples [21]. Broad Exception Suppression, the swallowed-error family, was the analyzer's most frequent check, at 263 findings against 185 in Study 3 [15]. That ratio is 1.42x, below the 1.80x headline [20]. AIRA also reports that an LLM evaluator misses these suppressions at 44:1 against the deterministic scanner, and states twice that being flagged is not the same as being a defect [16][17].
So the two results agree about how detection behaves. The deterministic scanner is the cheap way to produce candidates, and the verdict is a separate review step. I would still put the try/except rule in CI, as a candidate list attached to review.
For the zero-true-positive count to transfer, your corpus would have to resemble this one: short failure-path functions and pure-computation controls generated by Qwen2.5-Coder 1.5B at temperature 0.7 on CPU, 100 of the 120 samples on the failure path [1][2]. Your rule would also have to be scoped past try/except. This one stopped at try/except [6]. The independent unit is the task, so the effective sample is 12 tasks on one model family, adjudicated by one rater [11][22].
All 150 generations, including 30 from a 7B robustness footnote, are frozen unfiltered in the repo [3]. Generation ran once and is nondeterministic; what reproduces is the analysis layer, same 120 files in, same distributions and candidate counts out [13]. The legitimate-versus-swallow labels are human adjudication, published as gt.csv [4].
What to watch
- Whether a rule extended to if-guard default returns changes the candidate count on the same 120 frozen files.
- A second adjudicator scoring the gt.csv labels, which would test the single-rater caveat.
- Whether AIRA's C03 counts survive peer review once the preprint is published.