Build1 publisher3 min readPublished
1,254 dead mutants, a 100% score, and a payment charged twice
quayside shipped 1.0.0 with every mutant killed and 26 review findings triaged, then ran a payment twice on the dropped-connection retry it exists to prevent.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction

What happened
- quayside is an idempotency library for Node.js with a single-sentence API, execute(key, fn), which runs a function exactly once per key: if it already ran you get the stored result, if it is running you do not run it again. It has pluggable storage (memory, Redis, Postgres, MySQL), HTTP adapters for Express, Fastify, Hono and NestJS, and zero runtime dependencies. It shipped at 1.0.0 last week.
- Before release quayside achieved a 100% mutation score across 1,254 mutants, with every mutant dead and none suppressed; when a mutant was equivalent the rule was to delete or restructure the code rather than annotate it away.
- All quayside storage adapters pass one shared contract suite against real servers via Testcontainers, including 50-way concurrency races, SIGKILL crash recovery, and split-brain fencing where a stale holder must be rejected by the store itself.
- An adversarial review pass on quayside produced 38 candidate findings, of which 26 survived verification, and the ten worst were fixed before the 1.0.0 tag.
- The author's account is titled: my idempotency library had one job, and a dropped connection made it run the payment twice; the bug broke the library on the one scenario idempotency libraries exist for, the dropped connection followed by a retry with the same Idempotency-Key.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
quayside, a zero-dependency idempotency library for Node.js, shipped 1.0.0 last week behind a 100% mutation score across 1,254 mutants with none suppressed, a storage contract suite run against real servers, and an adversarial review whose 26 verified findings were fixed or triaged [1][2][3][4]. Its author then reports that a dropped connection made it run a payment twice, which is the single scenario the library exists to prevent [5].
The core design is legible enough to reason about. The atomic create-if-absent write is the lock: `execute(key, fn)` writes an `IN_PROGRESS` record before the function runs, so there is no window between checking and locking [6]. Success moves the record to `COMPLETED` and replays it for a result TTL; failure deletes the record so retries run fresh; a crashed process is covered by the lock TTL [7]. Every transition out of `IN_PROGRESS` is guarded by a fencing token validated inside the store itself, Lua on Redis and token-conditional UPDATEs on SQL, so a holder that lost its lease gets a `FencingError` rather than overwriting a newer result [8].
The defect was not in any of that. It was four lines in the Fastify adapter, written days before the tag to fix a different finding, and it survived until the final security pass [9][10]. Fastify is hooks rather than a wrapping function, so the adapter bridges the engine across two of them: `preHandler` takes the lock and stashes a deferred, `onSend` resolves it with the captured response [11]. The review found that `onSend` is not guaranteed to run: `reply.hijack()` skips it, and so does a handler that never answers, leaving the deferred unsettled and the key locked until the lock TTL expires [12]. In the shipped configuration example that is 30 seconds of 409s per retry with nothing actually executing [13][14]. The fix was to settle the capture on connection close, since Node's raw `ServerResponse` emits `close` for every request regardless of what the framework lifecycle does [15].
The post as published stops mid-patch, so take the chain as inference rather than the author's own words: a dropped connection is a closed connection, and the library's failure path deletes the record so retries run fresh [7][15]. A retry arriving with the same key after that then finds nothing, and charges again [5].
What the metrics measured is worth being precise about. The 1,254 mutants were mutants of code that existed when the suite ran [2]. The 50-way concurrency races, SIGKILL recovery and split-brain fencing tests belonged to the storage contract suite, not to an HTTP adapter's hook bridge [3]. The adversarial pass produced 38 candidates, of which 26 survived verification and the ten worst were fixed before the tag, leaving 16 verified findings unfixed at release [4][16]. None of those instruments has an opinion about a state machine spread across two framework hooks and a socket event, because that failure is an ordering of events, not a mutated line.
The transaction to note is the trade the remediation made: an availability bug, where a stuck key returns 409 and the caller waits out 30 seconds, became a correctness bug that moves money [13][5]. In payments those are not the same severity, and a fix applied days before a tag gets a fraction of the scrutiny the code it patches already received.