Build1 publisher3 min readPublished
Budgeting 57 hours for 2,000 test cases leaves 2.3 minutes per manual case
One Salmon product's 2,000 regression cases were planned at 57 hours across eight QA engineers and took five days. The proposed fix ranks changes by blast radius, using a formula that divides by a safety net its own worst case scores at zero.
The Engineer · Build desk
What happened
- One Salmon product carried 2,000 regression test cases, about 500 of them automated and 1,500 manual.
- Split across eight QA engineers, the plan put a full regression pass at roughly 57 hours per release.
- In practice every release took five days of full-team regression testing.
- The post proposes triaging each change before the regression pass using Risk = (Impact x Blast Radius) / Safety Net.
- It puts AI's role at compressing the map of what a release touches to fifteen or twenty minutes, with the risk decisions left to people.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost An estimate built from execution time alone hides the analysis, retesting and coordination hours, and those hours come out of the release calendar rather than the QA plan that was signed off.
- decision Once a change is scored high risk with no safety net, someone has to either hold the release or ship a known gap, and the coverage percentage no longer settles the argument.
- capability If the impact map really lands in fifteen minutes, QA can be consulted while the branch is still open.
- exposure The under-tested areas are the ones nobody flagged as different, so a healthy coverage number and a thin safety net on shared code can sit in the same release.
Divide the plan by the work it covers. Fifty-seven hours is 3,420 minutes. Set the 500 automated cases aside, spread the rest across 1,500 manual ones, and the budget allows about 2.3 minutes per manual test case [19]. The post says as much itself: fifty-seven hours "assumes pure execution: no analysis, no retesting, no coordination" [4].
Fifty-seven hours across eight people is 7.1 hours each, roughly one working day [20]. Read as a team total with an eight-hour day, the five days a release actually took is 320 person-hours [21], about 5.6 times the plan [22]. Part of the difference is the product shape: several connected products linked by cross-product flows, where a defect in one small piece could reach services that never touched the original change [5].
The triage rule is Risk = (Impact x Blast Radius) / Safety Net [9]. As three columns to sort a diff by, it is sensible. Blast radius asks how far the change reaches; safety net asks what already catches it [10]. As arithmetic it fails on the post's own worst case. The broken validation rule on a bank-details form is scored high impact, wide blast radius, and no safety net [12]. So the formula's worst case divides by zero and produces nothing [24].
Blast radius is the term that changes behaviour, and the diff does not carry the information it needs. The post's example is shared authentication: "A one-line fix to shared authentication code has a much bigger blast radius than it seems" [15]. Ranking that above a typo in a page title [14] requires knowing which callers reach the changed function. On the bank-details form, the validation logic runs on nearly every form in the app [12], and a reviewer learns that from the codebase.
The fifteen-to-twenty-minute figure for mapping a release [16] is the one most likely to be quoted and the least likely to transfer. It assumes two inputs: the full release-branch diff, so the model reads everything that changed instead of one file at a time, and ticket descriptions that carry business context [17]. Where tickets read "fix login bug", the model has the diff and nothing else to rank by. The worked release is also the easy case, a rewritten transfer flow and a changed card issuance fee calculation [18], both of which touch money.
The premise underneath all of it is that change volume is outrunning QA capacity. The post puts commit volume up roughly a quarter year over year in "a lot of orgs" and does not name them [8]. I would not plan against that figure. The five-day measurement stands on its own as the stronger evidence: 2,000 cases across eight engineers, five days per release [1][3]. As the post puts it, "Test case count was never the measure of quality" [7].
What to watch
- Regression elapsed time per release after the triage change, measured the same way the five days was.
- Tooling that supplies a real call map for blast radius instead of inferring reach from the diff alone.
- Whether Salmon ever holds a release on a high-risk score, the first point at which the scheme costs something.