Build1 publisher3 min readPublished
Grepping your tool for the error text a model promised catches an invented fix
A writer of reporting-tool cookbooks put hundreds of developer questions to current chat models and then tested the answers in a lab. The failures he describes cluster into five patterns. Validators clear most of them.
The Engineer · Build desk
What happened
- A writer of technical cookbooks for reporting tools says he put hundreds of real developer questions to current chat models. He then tested what came back claim by claim, running every recipe on a live system.
- He traces the first failure to the training corpus, where the most repeated answer wins and popular questions with no accepted answer are full of confident attempts that never worked.
- The costly class is gap-filling that parses: a plausible property name, or an XML element that is legal syntax in the wrong place. The tool accepts it and ignores it without emitting anything.
- He writes that most of the expensive failures he has found were accepted by every validator in the chain and wrong on the page, because parsers and validators check shape and not meaning.
- For a JasperReports book he started from forum questions with hundreds of views and no accepted answer, put each one to a chat model cold, and ran every checkable claim in a lab.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Each silent acceptance costs an afternoon of debugging pointed at the wrong component, and the developer who pasted the answer pays it.
- constraint A green validator stops being evidence for this class of bug, so the last gate available is a person comparing generated output against what the fix was supposed to change.
- capability An answer becomes falsifiable before any code is installed: the diagnostic it promises either appears in the product's own strings or it does not.
- decision Anyone weighing the per-question cost of this checklist has no published failure rate to weigh it against, so the adoption call rests on how silent their own toolchain is.
The model does not keep one release of a product separate from another. A default that flipped, an element that moved between namespaces: the answer that comes back is often correct for a release you are not running [7]. The check the post recommends is to state the exact release you run and ask whether the behaviour differs across versions, treating a vague answer as drift [6]. Then read the changelog yourself. "The changelog is the primary source; the model is a summary of summaries," the author wrote [6].
The cheapest check inverts the question. Ask what you would see if the answer were wrong, then confirm that the error it names exists in your tool [5]. A model that invents a fix will also invent the diagnostic, the author wrote, and a diagnostic string you cannot find in the product did not come from the product [5]. It works because there is no compiler, no report server and no database behind the model, and it has never seen the error text your tool emits [9]. When it promises you an exception, it is predicting what exceptions usually look like [9].
The shape of your question matters too. Put an assumption in the question and the model treats the assumption as a fact [4]: ask why the total breaks when you set the aggregate property, and it will explain at length why the aggregate property breaks totals, even when the aggregate property is not the cause [4].
Two of the checks require running something. Strip the problem to the smallest file that shows the symptom, apply the fix, and run it before touching the real report. If the smallest case does not change, the fix does not work, whatever the explanation sounded like [10]. Then diff the output, count the rows, and open the generated file to look for the thing the fix was supposed to put there [11]. "Silent success and silent failure look identical from the outside," the author wrote [11].
The rest is reading: the schema, the vendor's own reference for your version, the library's source code [12]. A second forum answer does not count as confirmation, because it draws on the same corpus the model learned from [12]. The post does not name the models tested or report how many of the hundreds of answers failed [15]. The JasperReports sample was selected for difficulty, since every question in it had already failed to attract an accepted answer [14]. I would not read a failure rate for ordinary questions off a sample chosen that way.
The post lists five causes and six checks [1]. Four of the six are asking and reading; two put code on the version you actually run [2].
What to watch
- If the author publishes per-model results from the claim-by-claim tests, the pattern becomes measurable instead of descriptive.
- Reporting tools that log ignored properties instead of dropping them silently would remove the costliest class from this list.
- Vendor changelogs published per version in machine-readable form would cut the manual release-note step out of the version check.