Build1 publisher3 min readPublished
A candidate model's 1.6-point accuracy gain lands inside both confidence intervals
A dev.to post's worked upgrade table gives the new model a 1.6-point accuracy edge its own intervals cannot support, while p75 latency doubles and median output tokens climb from 180 to 410.
The Engineer · Build desk
What happened
- A dev.to post argues that swapping a model ID when the sunset notice arrives is a common way LLM-backed systems degrade, because "the model works" is a separate claim from whether the model is safe in production.
- The post's account of what actually happens is that evaluation gets skipped, then run in the last week before the sunset, in a panic, on whatever inputs someone had lying around.
- It sets out a worked upgrade table comparing current and candidate models on one frozen dataset with the same prompt, across accuracy, latency p50 and p75, and output tokens at the median.
- Its stated rules are twenty runs per model to get intervals, p75 instead of mean latency because that is the request a quarter of users wait for, and a hold on any dimension that regresses.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Output tokens compound over every request, so a provider's lower per-token price can still arrive as a larger monthly bill for whoever owns the inference line.
- constraint The provider sets the sunset date, so a suite with a three-day turnaround cannot be written, frozen and run twenty times inside the window it leaves.
- decision Shipping the candidate in the post's table means trading a weak-field gain against a doubled p75 wait, and someone has to own that call in writing.
- exposure Spot checks land on easy cases, so the inputs where a new model changes its mind reach users unmeasured.
Subtract the two overall accuracy numbers in the post's table and the candidate is ahead by 1.6 points, 91.2% against 92.8% [5][1]. Now look at the intervals. The current model's span is 89.0 to 93.4 and the candidate's is 90.7 to 94.9, so the two share the band from 90.7 to 93.4, 2.7 points wide [5][2]. The post reads that as noise and says no improvement has been shown [10]. Apply the same test to the weak field, which the post treats as the real gain: 74.5% [70.1 to 78.9] against 81.0% [77.0 to 85.0] is 6.5 points, and those intervals also overlap, from 77.0 to 78.9 [6][9][3].
Latency moves one way. The median goes from 1.4s to 2.6s, up 86%, and p75 goes from 1.9s to 3.8s, exactly double [7][4]. The post's general case is a median moving from 1.5 seconds to 3 seconds, which it calls easy to miss in a manual test and hard to miss in a UI that blocks on the response [12].
Median output tokens go from 180 to 410, which is 2.28 times as many [8][5]. Output tokens are the expensive ones and they compound, the post says, and it offers the case of a model priced 20% lower per token that produces 80% more of them [11]. That multiplies out to 0.8 x 1.8 = 1.44, a 44% larger output bill for the same task [6]. Run the table's own ratio through the same calculation and the candidate has to cost 44% of the current per-token price, a 56% cut, before the output line stops growing [7].
The table in the post is an illustration, not a measurement published from a named production system [9]. Three conditions have to hold before numbers of this shape transfer to your workload: an identical prompt on both models, one frozen dataset across runs, and inputs drawn from your own traffic [9][13]. The post's case for the third is that reported gains "were measured on public datasets that look nothing like your contracts, your invoices, or your support tickets", and that "better on the benchmark" and "better on your data" are correlated, not equivalent [16].
Casey et al. report a deprecation and migration cycle of roughly twelve months, hitting every product built on the retired model, and they call task-specific evaluation essential while ruling manual evaluation too slow and too expensive for the migration window [2]. The post states the process problem directly: "Few people skip a test suite that runs in a minute. Many skip the one that takes three days." [4] Pair that three-day turnaround with the post's own recommendation of twenty runs per model and a serial evaluation takes 60 days [13][8]. The runs have to be parallel, and the dataset has to predate the sunset email. For any dimension that regresses, the post's rule is that the answer is "not yet", not "no" [15].
What to watch
- Whether the roughly twelve-month cadence reported by Casey et al. holds through the next round of provider sunsets or compresses.
- Whether providers start shipping deprecation notices with per-task regression evidence instead of public benchmark deltas.
- Whether anyone publishes a table of this shape from a named production system rather than as an illustration.