Anthropic's CI jobs grew two and a half times as fast as its test suite

Jobs grew two and a half times as much as the suite over the same six months (25 divided by 10) ^[10]. The suite got bigger, and the number of invocations got bigger faster. Turning either multiple into a budget line for another team's cluster would take what the dev.to writeup leaves out: absolute test counts, baseline job volume, and what the CI bill is ^[17].

Package relevance and historical performance are the two inputs to the selection service ^[4], and each carries a prerequisite. Package relevance needs a dependency graph with boundaries fine enough that a two-line change resolves to a small set of packages. Historical performance needs per-test records accumulated over time: runtime, pass rate, flake history. A single-package monolith with no per-test history fails both.

The alternative is letting a model pick. Asking an LLM which tests to run "isn't a verification strategy, it's a guess with good grammar," the post's author wrote ^[7]. In my view that is the right call here for one specific reason: a selector's mistake is silent.

If the suite is the signal the agent steers by ^[5], its latency bounds how many correction cycles the loop completes before someone intervenes ^[15]. Answering a two-line change with a full regression run spends compute and the feedback window the agent needs to fix its own work ^[11].

Flaky failures are the awkward case for that loop. A human sees red and judges it unrelated to their change, usually correctly, because they carry context the test output does not contain; an agent has none of that unless it is handed over explicitly ^[12]. The people positioned to supply that context are the same ones approving a significant share of pull requests ^[6].

The post's throughput example is a developer who shipped five meaningful changes a sprint and ships twenty with an agent, four times as many ^[8][9]. Pull requests, code paths, test executions, review load, deploys, and the number of failures somebody investigates all rise with it ^[8]. Code review, CI capacity, observability and deployment controls still run at the speed they ran before generation sped up ^[14]. According to the post, more automation at Anthropic kept control where it was and handed the control system more volume ^[16].

Agents write tests quickly too, and the post is blunt about what that produces: twenty tests covering variations of the same behavior while missing the failure mode that would page someone at 2am ^[13]. Test count and test quality are separate things, and the post calls "we generated 5,000 more tests" a terrible metric ^[13].