26 tests where a senior engineer writes 9: the calibration problem in AI test generation

An engineering team at KushoAI has published an account of what went wrong when it pointed large language models at API test generation, and the headline number is a ratio, not a throughput figure: on one endpoint the model produced 26 tests where a senior QA engineer would have written 9 ^[1][17]. That is roughly 2.9 times the human baseline for the same endpoint ^[18], and the interesting part is that the extra tests did not buy coverage. On a second endpoint the model missed the boundary cases that mattered most ^[2]. On a third it wrote tests that confirmed a response arrived and said nothing meaningful about what the response contained ^[3]. The team's summary is blunt: the tests were valid, the test suite was not useful ^[4].

Running the pipeline across thousands of real APIs surfaced six recurring failure modes ^[5], and read together they describe a calibration problem rather than a knowledge gap ^[12]. Over-generation is the visible one: 20 to 30 cases where 8 would do, with the excess plausible but redundant, which makes a suite harder to review and trust ^[6]. Under-generation is the one that actually costs you, and the team calls it arguably more dangerous, because omissions such as missing required fields, invalid enum values, boundary conditions, malformed formats, absent authentication cases and expected error responses are hard to spot precisely because they are absent ^[7]. Redundancy hides in phrasing: three tests labelled "invalid value", "incorrect value" and "unsupported value" often collapse to one check ^[8]. Inconsistency means two structurally similar endpoints get different treatment run to run, one receiving thorough positive, negative and boundary coverage and the other five mostly happy-path tests ^[9].

The two subtlest failures are the ones that reach production. Weak assertions check that a response matches a broad schema shape while never checking whether a transaction ID is absent when a payment is rejected, whether an error code is machine-readable rather than a human string, or whether the rejected state is reflected consistently across nested fields ^[10]. Cross-field relationship failures are worse: valid amount, valid currency, valid payment method, invalid business state, which returns a 200 with corrupt data three levels deep instead of a 500 ^[11].

Prompting is the obvious first response, and according to the KushoAI account it helps but plateaus, because the hard cases need judgment rather than more instructions ^[13]. The architecture the team argues for splits the two jobs: a fine-tuned model decides what deserves testing, and a frontier model writes the executable payloads and code ^[14]. In that split, retrieval fills in missing information about the system under test while fine-tuning calibrates QA behaviour itself, meaning prioritisation, redundancy reduction and assertion quality ^[15]. Execution feedback closes the loop, with accepted, rejected, edited and flaky tests routed back into the judgment layer ^[16].

For anyone buying or building this, the operational implication is that suite size is a vanity metric and review cost is the real one: a 30-test endpoint suite that a human has to triage is a transfer of work, not a saving ^[6]. Worth watching is whether the vendor publishes what the loop produces, because the material as published stops at diagnosis and architecture and offers no before-and-after acceptance, edit or rejection rates for the fine-tuned judgment layer ^[19]. The measure to ask for is not tests generated per minute but tests kept per endpoint, and whether the boundary and cross-field cases show up without a human adding them ^[7][11].