A Second Model's Opinion Is Not Evidence. A Mismatch List Is.

A walkthrough published on dev.to makes a narrow, testable claim about reviewing AI-generated code: asking a second model to review the diff gives you an opinion, usually in confident wording, and not evidence about runtime behaviour ^[1][2]. Its alternative is to deploy the candidate to a shadow service, replay a day of real traffic against it, and compare what comes back ^[3].

The distinction that matters here is reproducibility, not sophistication. Comparing status codes and bodies from two implementations answering the same request should produce the same mismatch list on a second run over the same log, apart from non-deterministic features you deliberately normalise ^[4]. According to the author, a model-generated review is harder to reproduce because temperature, prompt wording and recent context change the output ^[5]. One artifact can be handed to a colleague who disagrees with you. The other has to be re-argued.

The failure class this targets is the one your suite already passed. The post's example is a patch that compiles, passes the function suite, and still breaks on the sixth request in a burst because it assumes the items array is never empty ^[6]. Static review catches style and obvious logic errors; it does not see the request pattern your service actually receives on a Wednesday morning ^[7].

The harness is intentionally simple ^[9]: read a JSONL request log, call both baseline and candidate for each entry, then print the mismatch count and the first twenty records, each carrying path, method, both status codes and both bodies ^[8]. Note what is switched off by default. Write methods (POST, PUT, PATCH, DELETE) are skipped, because replaying writes safely requires a shadow database and careful cleanup ^[10][11]. Authorization, cookie and x-api-key headers are filtered out before forwarding ^[12]. Each call carries a five-second timeout, and HTTP error responses are captured as a status and body rather than raised, so a 500 on one side against a 200 on the other lands in the mismatch list instead of stopping the run ^[13][14].

Two costs are worth pricing first. Every replayed request becomes two requests, one to each service, so a day of read traffic doubles against whatever those services depend on ^[15]. And the shallow body comparison in the example is not usable as shipped: the author says to replace it with a canonicaliser that strips volatile fields such as IDs, timestamps, trace IDs and generated URLs, or the mismatch list will be noisy enough to ignore ^[16]. The canonicaliser is the actual work. The HTTP plumbing is not.

The shadow target does not need to be production-grade, only close enough in dependencies to respond realistically ^[17], which is why cheap disposable infrastructure changes how often you verify rather than what the verdict is ^[18].

The model keeps a job, just not the deciding one. After the deterministic compare, the post suggests using a model to group mismatches into a shorter report, for example 12 timestamp-only, 3 empty-array handling, and one 500 on a missing locale ^[19]. In that illustration, 12 of 16 mismatches, 75 percent, are normalisation debt rather than behaviour change ^[20], which is itself the argument for building the canonicaliser before you build the summariser.

What to watch: whether your log capture retains enough of each request to replay it at all, since the harness expects path, method, body and headers per line ^[8]; and what you decide about write paths, because a read-only replay leaves the riskiest changes unverified ^[10]. The version of the post supplied here refers to a decision table for grading differences but cuts off before showing it ^[21], so the triage policy is still yours to write.