Loading today’s stories
Build1 distinct publisher3 min readUpdated
A budget guarantee with 147 passing tests overran by 303% in production. The defect was already written down in three places, none of which ran against the deployment.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Twenty-five serverless instances, each enforcing a $0.05 ceiling perfectly against its own slice of traffic [6], is a $1.25 ceiling [1]. No line of code contains that number, which is why no unit test could assert against it.
The measured overrun was 303% [4], not 2,400%, because fan-out depends on load. Twelve workers spread across fewer instances and passed on some runs; twenty-five made the failure deterministic [11]. Concurrency in the scenario is not a stress level, it is a device for reaching more replicas. Of those 25 workers, 20 were admitted against a budget with room for two [5], ten times the intended admissions [2].
The earlier version of the same scenario is the more useful artifact. It asked the service how much it had spent and was told $0.0084 against a $0.05 ceiling [10], 17% of budget [4], while the deployment was 233% over it [10]. Both reads landed on an instance that had served a single request [10]. Spend is now computed as admitted calls times observed cost per call, the arithmetic a provider would bill; the ledger's own figure is recorded next to it and the disagreement between the two is the finding [12].
Which leaves the awkward part. The store's docstring said it was "correct for one replica and useless for two" and named that as measured failure #1 [7]. The deployment doc said not to run it behind a load balancer [8]. The baseline benchmark had already counted the single-replica ledger as the first of four ways budget enforcement fails open [9]. Three accurate descriptions of the defect, none of them executable against the running URL [20]. Documenting a failure mode and controlling for it are separate pieces of work, and the second one is the one that ships.
The harness charged its own price: one real defect to four false accusations [13]. It invented routes at /work, /about and /writing, got a 404 and declared the site broken, so routes now come from sitemap.xml [14]. It read 2 of 34 citations as unresolved because a tag-strip did not decode apostrophes written as HTML entities [16]. It reported a side channel by demanding byte-identical responses to "no token" and "unknown token", when an attacker already knows which one they sent; the property worth checking is that two rejected credentials are indistinguishable, since verified against five guessed principal names [17]. Each correction is recorded in the code rather than quietly amended [18]. Eighteen scenarios now pass [19].
The most interesting false accusation was the one that found a stronger system than the author expected: citations are not links to scrape but a build-time allowlist of pre-verified passages the chatbot may only quote from, so verifying the allowlist verifies every citation the site can ever emit [15].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
Four deployed products each had a fully passing unit suite: 147 tests on the portfolio, 77 on the budget proxy, 53 on the conformance checker, 78 on the retrieval system.
The scenario rules were that each scenario had to be something a person would plausibly do described in their words, and that no scenario may import product source code: it gets a URL over the public internet, the same as a stranger.
The author wrote scenarios that hit the live URLs instead of the code, and one product was overrunning its central guarantee in production.
The measured overrun of the spend ceiling in production was 303%.
Twenty-five concurrent workers were run against a budget with room for two, and twenty were admitted.
The cause was 25 serverless instances, each holding its own in-memory ledger and each enforcing the $0.05 ceiling perfectly against its own slice of traffic, giving 25 ceilings instead of one.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed first-party measurements, no external check
The account is unusually specific for a personal blog: named test counts, worker counts, admitted/refused splits, a quoted docstring, exact reported spend ($0.008400 against $0.05), two overrun percentages, and a coherent mechanism (per-instance in-memory ledgers under serverless fan-out) that explains both the breach and the false green self-report. It also volunteers disconfirming detail — four of five findings were the author's own harness bugs — and points at a scenarios directory for reproduction. What is missing is anything outside the author: one publisher, one item, no named platform or provider, no third-party reproduction, and no post-fix re-measurement of the ceiling itself.
One practitioner, four self-owned deployments
Observed uptake is confined to the author: four live deployments he owns, one scenario suite of eighteen passing public-URL scenarios, and one measured production breach. No other team, product, employer or tool is reported to have adopted the black-box scenario practice, and there are no download, usage, customer or third-party deployment figures in the cluster.
Understated relative to its own evidence
The headline number (303%) is dramatic, but the post immediately discounts itself: it reports that four of its five findings were false accusations produced by the harness, names each error, notes that a 12-worker run passed intermittently, and confines its claims to cent-scale budgets on the author's own projects. The generalisable finding — replicated in-memory enforcement plus self-reported telemetry defeats a green suite, and the defect can be documented in three places that never touch the deployment — is stated more modestly than the evidence presented would allow.
Self-published portfolio credibility
This is a self-published post by the owner of the products under test, and the products are portfolio pieces: one retained scenario exists specifically to keep newly added portfolio projects findable in the sitemap, chat corpus and MCP tool list, and the post closes by linking its own scenarios directory. That creates a visible reputational incentive to look rigorous. Offsetting it, the author discloses a defect in his own shipped guarantee and attributes four of five findings to his own harness, which cuts against pure self-promotion. No sponsorship, vendor relationship or commercial disclosure appears in the cluster.
Coherent but single-source and self-reported
Internally the account hangs together — mechanism, numbers and the two overrun figures are mutually consistent, and the author's self-corrections make deliberate inflation less likely. But the entire cluster is one dev.to post by the operator of the systems under test, with no named platform, no independent reproduction, no confirmation that the distributed-ceiling defect was fixed, and no evidence that anyone else has run the practice. Confidence is therefore moderate in the narrative and low in any generalisation beyond this author's deployments.
build
Three manual interventions in a month, and every guard was working as designed1 distinct publisher
build
Six MariaDB versions, one real difference: the only reason to leave 10.6 is the July 2026 clock1 distinct publisher
build
Force the tool call, then hand Lightsail a long-lived key1 distinct publisher
build
AI-written code fails the same four ways, and every gate you own reports green1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 23, 2026