Build1 publisher3 min readPublished
Adding NATS JetStream to a Jepsen-style harness took zero changes to the checker
Faultline injects verified faults into etcd and NATS JetStream, records what clients saw, and searches that history for a linearization. It found no violations. The case for believing that rests on the bugs it found in itself.
The Engineer · Build desk

What happened
- Faultline is an open-source Go harness that injects faults into a distributed system, records every concurrent operation its clients observed, and checks whether that history is linearizable, following Jepsen's methodology.
- Its reference target is a three-node etcd cluster, with NATS JetStream's key-value store added as an architecturally distinct second target to show the harness is not etcd-specific.
- No consistency violation was found in either target under the tested conditions.
- The write-up rests its case on six bugs the harness caught in itself while being built, on the grounds that a correctness tool which has never caught anything cannot be trusted.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability With the injector, workload generator and checker sitting behind a four-method client contract, a team's second datastore costs one integration instead of a second harness.
- exposure Fault injection puts the harness inside the blast radius: when cleanup gives up partway through, the cluster stays in the injected state until a human works out why.
- decision Anyone publishing a no-violations result now has a bar to clear first, because an unvalidated checker and a correct system produce the same report.
- constraint The verdict binds only to the seeded schedules and operation mix that were exercised, and the write-up does not report the campaign count needed to size that coverage.
`checker.Check` runs a Wing and Gong style search over the sequential orderings of a recorded history, memoized on the pair `(remaining-operations, sequential-state)`. The search budget is bounded, so a history it cannot resolve returns inconclusive instead of hanging forever [8]. Two details in that code are the ones I would copy. Numeric equality is exact, through `big.Rat`, applied recursively across JSON-shaped values. A client whose wire protocol round-trips numbers through JSON turns a Go `int` into a `float64`, and a naive comparison flags every one of those operations as a false violation [9]. The memoization key is only a bucket hash, so every cache hit is confirmed with full structural equality before it is used [10].
Faults go through Apply, Verify, Clear, across partition, kill, delay, drop and reorder [20]. A partition's `Verify` pings across the intended break and asserts that the ping fails, and nothing counts as coverage until it is confirmed against the real container [7]. That check passed unconditionally for a while, because the container images did not ship `iputils-ping`, so `ping` exited with "command not found", which in code is indistinguishable from a partition genuinely blocking traffic [15].
`containerIP` read only Docker's legacy default-bridge field, `.NetworkSettings.IPAddress`. That field is empty for any container on a user-defined network, and every deployment here uses one. The partition injector silently resolved an empty peer IP until the first live test [13]. `Clear()` for partition, kill and netem faults returned immediately on its first error and abandoned cleanup for every other node [14]. That stranded a live three-node etcd cluster in a partitioned, unrecoverable state for several days before diagnosis, and cleanup is now best-effort across every node regardless of earlier failures [14]. The fourth documented bug is in the ToyKV fixture. The server wrote the CAS outcome to `wireResult.OK` while the client read `wireResult.Value`, so every real client saw `nil` for every CAS regardless of outcome. The unit tests exercised the store directly and never went through the HTTP path [12].
Three of those four are in the fault-injection and Docker plumbing; one is in a target's client [18]. The post counts six bugs caught in the harness itself, and four of them are documented [17]. Its author wrote that "a correctness tool that has never caught anything isn't credible" [5].
The clean result is evidence about the schedules that were actually run. The write-up describes the outcome as "no violations found across N campaigns, fully reproducible", and it omits N, the workload size and the fault mix [19]. Reproducibility is cheap here because the campaign package derives both the fault schedule and the workload from one seed, then checks the recorded history [21]. For the etcd verdict to carry to another etcd deployment, that deployment's clients would have to use linearizable reads and transaction-based CAS the way this target does [16]. Its fault schedule would have to reach the same five fault types [20].
What to watch
- Whether the repo publishes campaign counts, seeds and durations, so someone else can re-run the no-violations claim.
- Whether a target with a weaker consistency model gets added without touching the checker or the injector.
- Whether the two remaining bugs of the six get written up with the same detail as the first four.