Cost Per Accepted Result Is the Only Model Price That Means Anything

A dev.to writeup on post-eval routing argues that the decision after a model evaluation is not which model wins but which model gets which traffic, measured as cost per accepted result rather than price per token ^{[2] [5]}. That matters because every failure mode the author lists is invisible on a pricing page, and two of them are properties of your own code rather than the model's. The setup: a new model drops roughly every other week, each arriving with the same two claims, cheaper and better ^[1] - call it something like 26 release cycles a year to absorb ^[24]. According to the author, every time they responded to a passing eval by flipping a single config variable, one of three things happened within a month. The cheap model was not cheap on their workload, because it needed more retries, longer prompts, or produced output the downstream parser rejected. The strong model got spent on renaming variables and writing docstrings. Or a silent regression appeared in the task category with the fewest eval cases, and they found out because a coworker did ^[3]. The arithmetic is the whole argument. Cost per accepted result is raw token cost divided by acceptance rate ^[5], so a model that is 5x cheaper per token but passes your checks half as often is really 2.5x cheaper, before the engineering time spent babysitting retries ^[6]. The author's illustrative figure is a parser that rejects 12% of outputs ^[4]; at an 88% acceptance rate you are paying about 1.14x sticker, roughly a 14% overhead that never appears in a per-million-token comparison ^[23]. The instrumentation is deliberately dull: one JSONL line per attempt recording task type, model, prompt hash, input and output tokens, latency and an accepted boolean ^[7], then a weekly rollup grouped by task type and model that prints acceptance rate and call count ^[8]. Notably, the price table in the published code is left empty with a warning not to hardcode numbers you have not verified this month ^[9]. For middle-difficulty work the pattern is budget model first, escalate to the stronger model with the failure message included in the prompt ^[10]. Which puts the load on the acceptance check, and the author says so plainly: if you cannot write one for a task type, the routing decision for it is superstition, and keyword-and-length heuristics for summaries are named as the weakest check in the set ^{[11] [12]}. Three companion posts, all carrying a MonkeyCode product-outreach disclosure ^[22], describe what a weak boundary looks like. A quota hit truncates JSON, a proxy restart returns an HTML error page with the same 200, and a model can return valid JSON missing the one field your code reads ^[13]. A json.loads inside a try block catches neither ^[14]: when {"result": ...} becomes {"text": ...} after a model update, the lookup returns None and the code writes None into a production record ^[15]. The proposed contract probe checks transport, shape, cost and error behaviour, and fails with a reason so policy can tell a timeout apart from a schema change ^{[16] [17]}. It is explicitly not a quality evaluation ^[25]. A fourth post's parser tolerates extra keys but exits non-zero and prints nothing when a required one is missing ^[18]. Read those together and the coupling is uncomfortable: the acceptance rate in your routing log is partly a measurement of your validator's strictness. Tighten the schema and the budget model's measured cost goes up without the model changing. Worth watching: whether two to three weeks of real traffic is actually enough to fill the per-task table, which is the author's stated threshold ^[19]; and whether contract probes survive contact with CI, since running them on every build tests the provider's availability as much as your code, and flakiness trains people to ignore the signal ^[20].