Build1 publisher2 min readPublished
On ARM64 a single dsb sy drops an impossible result from 2.3% of iterations to zero
A two-thread litmus test hits an outcome no interleaving allows on 2.3% of a million ARM64 iterations. Blocking the compiler leaves 1.8% of them. The store buffer is at the centre of the case.
The Engineer · Build desk
What happened
- A two-thread litmus test writes one variable and reads the other in each thread, and no ordering of those four operations lets both loads read zero.
- Run on ARM64 for a million iterations with no barrier, that outcome showed up on 2.3% of iterations, roughly one in forty.
- Adding a compiler barrier, which emits no CPU instruction, moved the rate only to 1.8%.
- Substituting a real hardware fence, one dsb sy, took the rate to 0.0% and the outcome vanished.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision The 1.8% row says the cheap remedies do not fix this. Marking variables volatile or dropping in a compiler barrier leaves the defect where it was, so the fix has to change the instructions the compiler emits.
- exposure Only a second core can see the reordering, so a single-threaded test and a single-core runner will pass the code that corrupts data in production.
- constraint The 2.3% comes from one unidentified ARM64 machine, so nobody else should expect that rate; a team migrating has to run the litmus test on the hardware it is migrating to.
A store is not finished when the instruction retires. It goes into the store buffer and drains to cache later [2]. A load to a different address has no dependency the core can see, so the core lets that load pass the pending store and keeps the pipeline busy [2]. Executed that way, thread 1 reads Y before it writes X, thread 2 does the mirror image, and both loads return zero [4]. From inside either thread nothing looks different, and only a second core can observe the difference [5].
2.3% of a million iterations is about 23,000 hits [15]. The compiler-barrier run recorded 1.8%, or roughly 18,000, so blocking the compiler removed about a fifth of what the unbarriered run saw and the hardware produced the rest [9][16]. The run with a real fence recorded 0.0% [10].
Read 2.3% as a measurement of one machine running one harness. The post gives the ARM64 percentages but does not name the CPU, the core count or the kernel [17]. How often the outcome appears depends on how often one thread's load executes while the other thread's store is still sitting in the buffer [2]. That is a function of the part and of how the two threads get scheduled. I would expect the order of the three rows to hold on other ARM64 hardware, and I would not quote the 2.3% as anyone else's rate [8][9][10].
The harness declares X, Y, r1 and r2 as volatile int, and offers two barriers: mfence on x86_64 and dsb sy on aarch64 for the real one, an empty asm volatile with a memory clobber for the compiler-only one [6][7]. The volatile is there to stop the compiler deleting or moving the accesses, so the only reordering left to observe is the hardware's [13]. According to the write-up, volatile is not the right tool for real concurrency and C11 atomics are [13]. What the litmus result gives you is the layer the patch has to reach [13].
On x86 the same code behaves differently. TSO permits store-to-load reordering and nothing else, and the store buffer drains aggressively, so a short run often records zero hits and iteration counts have to go up before any appear [11]. The harness admits as much when it comes back empty: "try more iterations or a different CPU" [12]. The post opens with the report it says recurs, in different words, on every ARM migration: "the same code works on our Intel CI and on the developers' older MacBooks, but it corrupts data / deadlocks / returns impossible values on Graviton" [1].
What to watch
- Whether the CoreTracer results get published with the ARM64 part, core count and kernel behind the 2.3% run.
- A version of the same litmus test written with C11 atomics instead of volatile, showing how much ordering the fix actually needs.
- An x86 run with iteration counts high enough to register hits, which would put a number on how much cover TSO gives.