Reinjecting the bug is how this experiment scored its coding agent's patches

Putting the defect back is a test of the test. If the focused test stays green with the bug reinstated, it never covered the behaviour anyone cared about. "A green test that does not fail when the defect returns gives false confidence," the author of the dev.to write-up wrote ^[8]. That step comes after the two ordinary ones: the patch has to satisfy a behaviour contract read out of the source, and the focused test has to pass ^[7].

The LLM's role ended before the correctness call. It was used in an earlier round to help decide which questions were worth investigating, and kept out of the pass/fail call, because a model can prefer a short, plausible answer that names the wrong function ^[9]. For the change tasks, source checks and deliberate mutations were the final gate ^[10].

The scale is three conditions, five fresh agent sessions each, across three fixed change tasks in one private Python and TypeScript product ^[11]. That is fifteen runs per task if every task got a full set, and forty-five in total ^[17]. The post does not state the total.

Five successes out of five carries an exact 95% interval from 47.8% to 100% ^[14]. That lower bound is 0.478 to the fifth power, which comes to 0.025, so a tool that succeeds on 47.8% of runs still returns a clean sweep of five about one batch in forty ^[18]. Choosing an index on a 5/5 result means accepting a true failure rate that could sit at 52.2% ^[19].

The failure modes are more useful than the counts. On the hardest task the author expected the structured map to help most, and instead saw patches that looked plausible and were incomplete: a paginated field absent, a closed vocabulary changed incorrectly, an uncertain state left unprotected ^[13]. None of those facts live in an import graph. For that result to transfer to your repository, your unwritten contracts have to be the same kind, and your tasks have to cross a boundary the index does not model.

The comparison set was broader before it narrowed: ordinary source search, context packers, language-service bridges, graph indexes, and semantic search, which the author treats as doing different jobs, so that one headline score would hide more than it explains ^[20][15]. Two tools reached the change tasks, Code Review Graph and Serena, both of which give an agent structured help finding code relationships ^[11]. The write-up says neither stage is a universal ranking or a token-saving claim ^[22], and it describes the observations as small and task-specific ^[23].

The condition that shaped the whole exercise is stated plainly: "A short answer is a saving only when the resulting patch is correct" ^[4]. An index that finds four files fast and misses the fifth one carrying the contract converts saved tokens into rework ^[5]. The index is saved navigation. Correctness still comes from the source ^[6].