Skip to content

BuildNot yet confirmed elsewhere1 publisher2 min readPublished

Coding agents add mocks in 36% of test commits where humans add them in 26%

Hora and Robbes's MSR 2026 study of 1.2 million commits found coding agents add mocks in 36% of test commits, against 26% for humans. A dev.to post turns that gap into an argument for written mocking policy.

The Engineer · Build desk

How we use AISend a correction

Illustration accompanying Coding agents add mocks in 36% of test commits where humans add them in 26%
Generated illustration
Agents' test commits add mocks more often Figures from Hora and Robbes's MSR 2026 study of commits across public repositories, as summarized in a dev.to post.

Key figures: agent test commits add mocks about 1.4 times as often as human test commits. The study isolated 48,563 commits authored by coding agents, about 4% of the 1.2 million commits measured across public repositories.

Agents' test commits add mocks more often
FigureValueNoteClaim
Agent mock-adding rate vs human test commits, about1.4 times7
Commits by coding agents48,563 commits2
Agent-authored, about4% of commits8
Public-repo commits measured1.2 million commits1

What happened

  • Coding agents authored 48,563 of the commits, about 4% of the sample, and the study compared their test commits with everyone else's.
  • Agent commits cluster on a single test-double style, while human commits spread across stubs, fakes, spies and mocks.
  • According to the post, agent-generated suites also come out more brittle than the human baseline on the measures the paper reports.

Why it matters

  • decision A team that lets agents write tests has to choose whether to write down which test doubles are allowed, or leave the existing suite as the unwritten rule.
  • exposure Mock-heavy tests merged today become examples the next agent session copies, so repos with a long mocking history have the most patterns to amplify.
  • constraint The 1.4x ratio comes from public repositories, so it is a prompt to measure your own agent and human commits, not a forecast for your suite.

The gap is 10 percentage points, and the post's author is careful about what that licenses. The author wrote: "Ten percentage points on a measured dataset is a clear signal; it is not evidence that agents mock everything in sight." [6] As a ratio, agent test commits add mocks about 1.4 times as often as human ones [7].

The sample limits how far that ratio travels. Agents wrote about 4% of the 1.2 million commits [8]. The post does not say how many of those are test commits or which languages and agents the set covers. For the 36% to hold in your repo, your agents, your language and your mocking history would have to look like the public sample [3].

The causal story is the author's. The paper, as the post describes it, measured rates of test-double use and brittleness [1][9]. The remedy is the author's inference: the fix lives at the convention layer, because review cannot keep pace [12]. The mechanism offered is that the codebase is the specification. "Agents mock what the codebase mocks," the author wrote [10].

The case against review rests on an illustration. A reviewer who catches eight mocks in a diff of twenty tests has shipped twelve, and those twelve become examples the next session samples [11]. The numbers are tidy, as hypotheticals tend to be.

In our view, the remedy is cheap enough to try before the evidence is complete. The first step is to measure: the share of test commits that add a test double, split by agent and human authors. That is the comparison the paper made [3][4]. If your own gap comes out near zero, a policy can wait. If your suite is already mock-heavy and agents write tests at volume, we think the post's case holds, and a written convention gives the reviewer something to enforce.

What to watch

  • The paper itself (arxiv 2602.00409): its brittleness measures, languages and agent set decide how far the 36% travels.
  • Any team that publishes before-and-after mock rates for agent commits once a written convention is in place, since that would test the post's remedy.

Clarity's read

What the record supports and how the coverage leans. The claims behind it follow.

Reality

Evidence45
Adoption
Insufficient
Hype gap+25
Incentives
Insufficient
Confidence55
Why these scores

Claim ledger

Ranked by verification strength, evidence, and original report placement.

  1. [1]

    A 2026 paper by Hora and Robbes (arxiv 2602.00409, MSR 2026) measured 1.2 million commits across public repositories.

    ReportedSupportedSource: dev.to post summarizing Hora and Robbes, MSR 2026View cited source
  2. [2]

    The study isolated 48,563 commits authored by coding agents.

    ReportedSupportedSource: dev.to post summarizing Hora and Robbes, MSR 2026View cited source
  3. [3]

    Agent test commits add mocks 36% of the time.

    ReportedSupportedSource: dev.to post summarizing Hora and Robbes, MSR 2026View cited source

Sources

1 independent publisher whose own reporting we read for this story.

  1. dev.to

    1 article · October 12, 2026

    Over-Mocking Is a Governance Problem, Not a Review Smell

Share your take

Let Clarity write the post for you.

Signed-in readers get a short post drafted on this story in the register they choose — narrative, analytical, or a direct position — editable to the last word before it goes anywhere. The share buttons at the top of this story work without an account.

Topics and entities

Follow any of these and your For You feed starts watching them — no settings page required.

Topics

Entities

Loading related stories