Skip to content

Build1 publisher3 min readPublished

A balance-total test under concurrency catches ledger lost updates that unit tests miss

Developer sunny56's ledger-core harness runs 8 threads of 250 transfers each and asserts that its naive read-modify-write strategy loses money. The same balance-total check, run under concurrency, shows whether the locking strategies conserve it.

The Engineer · Build desk

Illustration accompanying A balance-total test under concurrency catches ledger lost updates that unit tests miss

What happened

  • An engineer writing as sunny56 says the payment bugs they chased mostly had correct formulas, correct rounding and passing tests, yet still lost money to simultaneous operations.
  • The author published ledger-core, a double-entry ledger with three concurrency strategies behind one interface and a test suite showing which ones conserve money.
  • The locking strategy sorts account IDs before taking locks to prevent deadlock, and a test with a join timeout fails the build if one occurs.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision More formula or rounding tests leave the lost-update class uncovered, so a ledger suite needs a many-writer test that compares the balance total before and after.
  • exposure Without that test, a lost update first shows up at reconciliation, after one account's balance has already been wrong for a day.
  • cost Teams that choose correct pessimistic locking pay in throughput on their busiest accounts, because every writer waits behind the hottest one.
  • constraint A green concurrency test is only evidence while a known-bad strategy still fails in the same harness, so the race window has to be kept wide enough to trip it.

The broken code is two lines: `var balance = store.Read(account).Balance;` followed by `store.Write(account, balance + delta);` [5]. Two threads read 1000 at the same moment. One writes 1100, the other writes 950, and the second write lands last, so the +100 transfer is gone [5]. Nothing throws. The author, who publishes as sunny56, wrote: "A lost update just makes the number slightly wrong, on one day, for one account, and nobody notices until reconciliation runs and somebody asks where four hundred dollars went." [2]

A formula test has one writer, so it cannot see this. The post holds the ledger to one invariant: the sum of all balances after N transfers equals the sum before. The author notes it is trivially easy to satisfy with one writer [4].

The NaiveStrategy test runs many writers at once. It uses 8 threads of 250 transfers each, all moving 100 from account 0 to account 1 [6]. That is 2,000 transfers on one account pair [1]. It asserts two things: no operation reported a failure, and the closing total differs from the opening total [6]. "Read that assertion again. It asserts the total changed. The test passes when money is lost," the author wrote [1].

Two details make it fail on every run. A Barrier releases every thread at the same instant. The naive strategy also calls Thread.SpinWait(50) between its read and its write, widening the race window on purpose [7]. "The window exists in real code too. It is just narrower, which means you hit it on a Tuesday in production instead of every time in CI," the author wrote [2].

I think keeping the broken strategy in the codebase is the best decision in the repository. It is a negative control. It shows the harness can detect a loss, and that is what makes its green result on the other strategies worth trusting. The author is specific about what to do if it stops failing: "I want them widening the window, because the test has stopped proving anything." [3]

The pessimistic strategy takes an exclusive lock on every account the entry touches and does the read-modify-write inside it. That is the SELECT ... FOR UPDATE shape [8]. The deadlock fix is one call. The store sorts account IDs with OrderBy before taking the gates, so a transfer from A to B and a transfer from B to A acquire locks in the same order [9]. A separate test with a join timeout makes a deadlock fail the build instead of hanging it [9]. The price is throughput: every writer queues behind the hottest account [10]. "If one account is popular, you have built a single-threaded system with extra steps," the author wrote [4].

The optimistic strategy snapshots each account's version, takes the lock, and applies the entry only if no version has changed. It retries up to maxAttempts [11]. The supplied text of the post ends inside that listing, before the apply step [12].

For a green result to carry over to a production ledger, the harness has to run against the store production uses. The test in the post runs against the harness's own store object [6]. In my view the conservation assertion also needs a second assertion that counts applied entries. If a transfer lost both its debit and its credit to overwrites, the total would come out unchanged.

What to watch

  • Published results for the optimistic version-check strategy under the same 8-thread, 250-transfer harness, including how often it exhausts maxAttempts.
  • A run of the conservation harness against a production database store instead of the harness's own store object.
  • Reports of the NaiveStrategy test starting to conserve on other hardware, the case where the author says to widen the race window.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories