The stopping problem: an LLM rewrite loop that converged on code javac rejected

An intern working on Pulsar Modernizer, a project meant to turn legacy Java into modern Spring Boot code, published what happened when he moved from the understanding half of the pipeline to the generating half ^[1]. His extraction component was already respectable, at F1 = 0.857 on an annotated corpus with a React front end running in Docker ^[2], and the write-up is useful precisely because the generation step is where the project stopped being a measurement problem and became a governance one ^[3].

The naive version failed inside an hour, and it failed in three ways worth separating ^[4]. Hallucination is the boring one: invented helper classes, method calls with wrong signatures, all of it caught by a compiler. The other two are not tooling problems. There is no stopping criterion, because the model announcing that it is done is not evidence that it is done ^[4]. And there is no equivalence criterion, because code that compiles is not code that behaves like the system it replaced ^[4].

The architecture he settled on is the correct shape for that problem. An APIDesigner agent emits a JSON contract with classes, methods, DTOs and declared exceptions; a CodeGenerator writes the .java; a TestGenerator writes the *Test.java; a verifier runs mvn test ^[5]. The load-bearing decision is that every rule extracted from the legacy code becomes a test the new code must pass, so the authority moves from the model to javac and JUnit ^[6]. He ran the whole thing on Qwen 2.5 Coder 3B via Ollama on a laptop, on the argument that a pipeline requiring GPT-4-level intelligence to hold together will not ship inside a bank ^[7].

Then the oracle bit back. He hand-wrote JUnit oracles for four rules from a banking corpus and got 2 of 3 green on a raw prompt and 3 of 3 after adding three conventions to the top of the prompt ^[8]. The conventions: null inputs are always rejected rather than silently ignored, "cannot exceed N" means N passes and N plus epsilon fails, and currency and country lists are case-sensitive ^[9]. Every failure he observed traced to one of those three, which means the defects were in his own rule annotations, not the model: grammatical English with no written tiebreaker, and a fluent model picking the other branch ^[10]. That is the finding operators should copy. A generation pipeline is an ambiguity detector aimed at your specification, and most legacy specifications have never been forced to disambiguate.

The repair loop is the cautionary half. Feed the compiler error back, up to five iterations ^[11]. On a simple currency-allowlist rule the model called .toUpperCase() on null, crashed at runtime, and fixed itself on the second pass ^[12]. On a velocity rule, more than 20 transactions in a sliding hour adding +30 to a fraud score ^[13], round one emitted a null check against a Java long primitive, javac refused, and rounds two through five returned character-identical code and the identical error ^[14]. Four of the five permitted iterations, 80 percent of the budget, produced no change at all ^[1]. His diagnosis is that the repair prompt was Markov-1, carrying only the previous attempt, so it sat on a fixed point ^[15]. A loop terminating tells you nothing about what it terminated on ^[16].

Two things to hold loosely. The author says outright this is a slice over four rules in a language with parsers everywhere, on a corpus he annotated himself, and that the results are directional rather than statistical ^[17]. And the reported tally is out of three while the oracle count is four, so one rule's result is unaccounted for in the numbers as published ^[2].