Build1 publisher3 min readPublished
Twelve of the 18 repos that broke their OpenAPI contract disclosed every break in the changelog
The scan's 95% figure covers 85 repositories, and replaying their release histories showed most projects announcing the breaks they shipped, while the authors' own date sort had invented about half the count.
The Engineer · Build desk
What happened
- Of the 85 repositories found publishing a genuine first-party OpenAPI spec, exactly four run any breaking-change detection, which the authors report as 95.3% with no compatibility gate.
- Fetching each project's spec at up to ten consecutive releases and diffing the pairs turned up 2,118 breaking changes across 60 repositories, 631 of them visible to consumers.
- The largest single case replaced every /v1/items route with /v1/entities in one release, removing fourteen paths and adding twenty-three, so any integration calling the old routes gets a 404.
- Fourteen releases across the whole scan shipped a consumer-visible break with written notes that never mentioned it.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Every working gate in the sample is oasdiff, and one of the four is oasdiff's own repository, so a team that wants this check is choosing that tool or building one.
- contradiction Most projects that broke consumers documented every break, so the fix the data supports is aimed at the quiet minority instead of at every project missing a differ.
- decision The price of the gate is a spec fetch and a diff per release plus a policy on which classes of change fail the build, and in return it catches the undisclosed break.
- precedent Any future repo-scan statistic that does not say how it ordered releases can now be challenged on this specific failure, because this one showed what the sort did to its own totals.
About half of the first count came from a sort key. A monorepo tags several components on the same day, core/v1.7.14 beside ent-v1.5.13-base, so walking releases in date order runs backwards through one component's versions, and every path the newer spec adds is reported as a removal [13]. One project looked like it had shipped 577 breaking changes. Restricted to a single coherent release line and ordered by version, it had shipped 249 [14]. Across the set, fixing the order took the total from 4,285 to 2,606 breaking changes and consumer-visible ones from 1,614 to 813 [15], so 1,679 breaking changes and 801 consumer-visible ones were artifacts of the sort [4].
Version order does not track release chronology either. One project shipped v0.38.26 in 2026, long after v1.0.0 in 2024, because 0.38 is a maintained LTS line, and diffing v0.40.0 against v1.0.0 compares two branches [16]. The authors now drop any pair whose newer release predates the older one. That removed 17 comparisons and took one repository to zero findings: a project they had ranked among their strongest examples had shipped nothing breaking [17].
Direction was the third bug. A request field becoming required breaks every client that omits it, while a response field becoming required breaks no client, because the server has promised to send more and the caller never had to supply it. Counting both as consumer-breaking cost 25 changes, taking the consumer-visible total from 813 to 788 [18].
The 95.3% figure describes 85 repositories, not 1,939 [2]. Those 85 were the ones with a genuine first-party spec, 4.4% of what the scan looked at [1]. Three of the 85 gate on Spectral or Redocly, which check whether the spec is well-formed [5]. A spec that has just deleted your endpoint passes both.
Eighteen of the 60 repositories with usable release history shipped a consumer-visible break, 30% of them [5]. The project that replaced its /v1/items routes marked the change with an IMPORTANT callout in its release notes, telling integration and software developers that the /v1/items* and /v1/locations* endpoints had been entirely replaced by /v1/entities* and pointing them to the entity merge documentation for the API changes [10]. The notes linked a migration guide and carried a backup warning [11].
Whether the 631 transfers to your project depends on who your consumers are. If they read release notes, compare against the disclosure rate. If they generate clients from the spec and pin them, a differ in CI reports a removal before the release goes out.
The authors wrote: "We are publishing the corrections because a statistic whose author never shows their errors is a statistic you cannot check" [19]. Their published totals of 2,118 breaking changes and 631 consumer-visible ones sit 488 and 157 below the last numbers the corrections log reaches [7]. One further correction, a keyword search of release notes for terms including "breaking" and "backwards-incompatible", is described without its effect on the count [20].
What to watch
- Whether the authors publish the repository list and diff output, so the 60 release histories can be replayed independently.
- Whether a second breaking-change detector shows up in any of these repos, given all four working gates today are oasdiff.
- Whether the remaining corrections account for the 488-change gap between the corrections log and the published totals.