Skip to content

Build1 publisher3 min readPublished

Adding NATS JetStream to a Jepsen-style harness took zero changes to the checker

Faultline injects verified faults into etcd and NATS JetStream, records what clients saw, and searches that history for a linearization. It found no violations. The case for believing that rests on the bugs it found in itself.

The Engineer · Build desk

Illustration accompanying Adding NATS JetStream to a Jepsen-style harness took zero changes to the checker

What happened

  • Faultline is an open-source Go harness that injects faults into a distributed system, records every concurrent operation its clients observed, and checks whether that history is linearizable, following Jepsen's methodology.
  • Its reference target is a three-node etcd cluster, with NATS JetStream's key-value store added as an architecturally distinct second target to show the harness is not etcd-specific.
  • No consistency violation was found in either target under the tested conditions.
  • The write-up rests its case on six bugs the harness caught in itself while being built, on the grounds that a correctness tool which has never caught anything cannot be trusted.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • capability With the injector, workload generator and checker sitting behind a four-method client contract, a team's second datastore costs one integration instead of a second harness.
  • exposure Fault injection puts the harness inside the blast radius: when cleanup gives up partway through, the cluster stays in the injected state until a human works out why.
  • decision Anyone publishing a no-violations result now has a bar to clear first, because an unvalidated checker and a correct system produce the same report.
  • constraint The verdict binds only to the seeded schedules and operation mix that were exercised, and the write-up does not report the campaign count needed to size that coverage.

`checker.Check` runs a Wing and Gong style search over the sequential orderings of a recorded history, memoized on the pair `(remaining-operations, sequential-state)`. The search budget is bounded, so a history it cannot resolve returns inconclusive instead of hanging forever [8]. Two details in that code are the ones I would copy. Numeric equality is exact, through `big.Rat`, applied recursively across JSON-shaped values. A client whose wire protocol round-trips numbers through JSON turns a Go `int` into a `float64`, and a naive comparison flags every one of those operations as a false violation [9]. The memoization key is only a bucket hash, so every cache hit is confirmed with full structural equality before it is used [10].

Faults go through Apply, Verify, Clear, across partition, kill, delay, drop and reorder [20]. A partition's `Verify` pings across the intended break and asserts that the ping fails, and nothing counts as coverage until it is confirmed against the real container [7]. That check passed unconditionally for a while, because the container images did not ship `iputils-ping`, so `ping` exited with "command not found", which in code is indistinguishable from a partition genuinely blocking traffic [15].

`containerIP` read only Docker's legacy default-bridge field, `.NetworkSettings.IPAddress`. That field is empty for any container on a user-defined network, and every deployment here uses one. The partition injector silently resolved an empty peer IP until the first live test [13]. `Clear()` for partition, kill and netem faults returned immediately on its first error and abandoned cleanup for every other node [14]. That stranded a live three-node etcd cluster in a partitioned, unrecoverable state for several days before diagnosis, and cleanup is now best-effort across every node regardless of earlier failures [14]. The fourth documented bug is in the ToyKV fixture. The server wrote the CAS outcome to `wireResult.OK` while the client read `wireResult.Value`, so every real client saw `nil` for every CAS regardless of outcome. The unit tests exercised the store directly and never went through the HTTP path [12].

Three of those four are in the fault-injection and Docker plumbing; one is in a target's client [18]. The post counts six bugs caught in the harness itself, and four of them are documented [17]. Its author wrote that "a correctness tool that has never caught anything isn't credible" [5].

The clean result is evidence about the schedules that were actually run. The write-up describes the outcome as "no violations found across N campaigns, fully reproducible", and it omits N, the workload size and the fault mix [19]. Reproducibility is cheap here because the campaign package derives both the fault schedule and the workload from one seed, then checks the recorded history [21]. For the etcd verdict to carry to another etcd deployment, that deployment's clients would have to use linearizable reads and transaction-based CAS the way this target does [16]. Its fault schedule would have to reach the same five fault types [20].

What to watch

  • Whether the repo publishes campaign counts, seeds and durations, so someone else can re-run the no-violations claim.
  • Whether a target with a weaker consistency model gets added without touching the checker or the injector.
  • Whether the two remaining bugs of the six get written up with the same detail as the first four.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories