Four test runs, a week's API budget: the seam that gets the model out of CI

The bill tracks your iteration loop, not the size of the suite. Four runs in one afternoon, with nothing broken, cost more credits than the author of the dev.to writeup had meant to spend across the whole week ^[1], which puts the average run somewhere north of a quarter of that intended weekly budget ^[13]. Debugging is the expensive case by construction: a flaky endpoint is the thing you rerun, and every rerun re-pays for every test that touches the model ^[2]. The alternative most teams pick is not a cheaper test but no test, leaving the model-calling paths uncovered and hoping, which is the other half of the bind the source describes ^[4].

The seam is worth cutting for the money, but the detail that decides whether you saved anything is the patch target. The router does `from services.ai_service import generate_summary`, so it holds its own reference to that function, and patching the defining module leaves that reference alone ^[10]. Get it wrong and the test still passes, because the real call still happens, quietly, at full price. A green suite is not evidence that the network stayed out of it. The usage dashboard is ^[1].

There is a cost to the stub as well, and it is not a testing cost. Patching the router's reference replaces the whole service function, so neither the prompt construction nor the `response.choices[0].message.content` unwrap is exercised by that test ^[14]. What the fast test proves is that the route moves a string from the service into a `SummaryResponse`, in milliseconds, deterministically ^[8][9]. The prompt and the parse are still checked only by production, unless you pay for a real call somewhere outside the per-commit loop on purpose.

What the boundary buys that credits cannot is the failure path. Set `side_effect` to an exception and the upstream timeout arrives on demand, and the 500 becomes assertable ^[11]. Timeouts, rate limits and malformed responses are the things that break in production and almost never get tested ^[11], and you cannot schedule a rate limit from a real provider for Tuesday morning. The same seam is what later makes a provider swap, a cache or a retry policy tractable ^[12], because exactly one module in the tree knows a model is involved ^[5].