Make the Model Earn the Traffic: Canary Suites Built From Your Own Commits

Three separate posts on dev.to describe the same defensive habit from different angles: a new model gets no production traffic until it has been replayed against the author's own recent work, graded mechanically, then sampled by hand ^[1][12][16]. That matters because the failures each author actually paid for are invisible in a leaderboard score ^[2][19].

The specifics are worth more than the principle. One author reports being burned twice: a "drop-in replacement" silently stopped emitting valid unified diffs, and a cheaper model passed all their prompts but tripled the retry rate on long files, which erased the savings ^[3]. A second says every hype-driven switch ended with a quiet revert about two weeks later, after the model mangled a migration script or confidently explained a bug that did not exist ^[13]. A third swapped a model in on buzz and it began translating variable names in generated code comments into different casing conventions; the docs site published subtly mangled API references for a week until a reader flagged it ^[17]. Their shared conclusion is that the dangerous release is the one that fails plausibly rather than loudly ^[18].

The extraction step is the part worth copying. The first author's script pulls roughly thirty commits touching `src/`, discards merge commits and anything touching more than four files, then uses the parent tree as the starting state, the human-written commit message as the instruction, and the real diff as the reference ^[4][5]. The argument for this over hand-written prompts is that you already know the tasks are representative, because they are work the project needed ^[6]. One hard precondition: do not point it at a repo with secrets in history ^[7].

Grading stays at three classes on purpose. Empty, unparseable, or malformed output that fails `git apply --check` is a hard fail; touching files outside the reference set is a soft fail for scope creep; everything else passes ^[8]. That check tests shape and scope, not semantic correctness ^[9]. The semantic pass is manual and deliberately small: open the candidate diff against the reference for ten randomly chosen tasks and ask whether code review would have caught the difference ^[10]. Two patterns are automatic disqualifiers -- silently dropped error handling and helpfully reformatted files -- and finding either twice in ten samples means no routing traffic regardless of the aggregate score ^[11].

The second author's variant is smaller and prompt-shaped: twelve YAML cases across untangle, write-under-constraint, diagnose, and translate, each with a mechanical check a machine can verify and a short by-eye list for the judgment calls ^[14][15]. Answers get written to files rather than printed, so two models can be diffed side by side, and temperature is pinned to zero with the caveat that zero is not guaranteed deterministic ^[15]. The third author front-loads an hour-long interview covering rigid format compliance, one subtly unanswerable trap question, a wander test for unrequested scope expansion, latency measured on their own prompt sizes, and one chore with a known answer; a single hard failure ends the tryout ^[16][20][21]. That author claims the hour rejects more than half of hyped releases, which is a single-sourced number from personal experience ^[22]. Survivors get a one-week observation period where the candidate answers every prompt into a log while the incumbent's answer is the one actually used ^[23].

None of this is free. Thirty tasks times N candidates, re-run at every release, is the real cost ^[24], and the second author is blunt that the expense is why most people outsource the decision to whoever tweets loudest ^[25]. The corpora described here are twelve to thirty items ^[26], which is small enough to run in an evening and small enough to miss things.

What to watch: whether the observation logs actually get reviewed on the scheduled day rather than accumulating ^[23], and whether anyone publishes rejection rates that hold up next to the informal better-than-half figure ^[22].