The only change to production you cannot roll back is the one your vendor ships

A pinned model string buys one thing, and it is worth buying: the date. It does not stop the behaviour underneath you from moving, because that change is not yours to make and usually not yours to roll back on your own schedule ^[1]. What pinning converts is the arrival channel. Without it you learn about the change from a support ticket ^[0]. With it you get a deprecation window, and a window is something you can put in a sprint. That is the entire case for treating the version string as release infrastructure rather than a line of config someone set once.

The reason average benchmark scores cannot do this job is arithmetic, not attitude. An average is one number over a distribution, and every property the dev.to piece names as breakable is a property of that distribution: how long answers run, how much the model hedges, which tool it reaches for first, what it does with a vague request ^[3]. A model can move all four and still score better on reasoning and on code ^[2]. Two models with the same mean are not interchangeable if one of them has started opening every reply with a summary that a downstream parser was never written to expect ^[5].

Which is why the frozen set has to be drawn from real traffic rather than written by hand. Well-formed questions are the easy ones to author, so that is what most eval sets contain, and that is exactly the population in which house style for filling gaps never gets exercised ^[7]. The malformed, half-specified requests are the measurement instrument here. Keeping them means keeping your own verdict on each output too, recorded while the old behaviour is still in front of you ^[8].

The diff is the cheap half. A diff reports what moved, not which side is better, and a person still has to read every changed case and rule on it ^[9]. That reading is the line item nobody plans for, and it scales with two quantities you do not control: how large your frozen set is and how much of it the vendor moved. The dev.to author's suggested starting point, twenty real requests with your honest opinion of each, written down before anything changes ^[13], is sized to be done rather than sized to be complete.

Twenty cases is a start, not infrastructure. Infrastructure means the frozen set is versioned alongside the code that calls the model, and each recorded verdict carries an author and a date, so that when the diff comes back you are comparing against a judgement someone owns instead of relitigating what the agent used to do. Once the upgrade is live, that argument is unwinnable, because the baseline you would need has already been overwritten by the thing you are trying to measure ^[11]. The source calls the surrounding machinery harness engineering ^[12]. The name matters less than where it sits on the org chart: this is the same category of work as your migration tooling, and it fails the same way, which is silently and in production.