Build1 publisher3 min readPublished
In One Example, 90ms of Clock Skew Made Last-Write-Wins Keep the Older Write
A dev.to walkthrough follows a reverted profile update back to a three-line conflict resolver, and the same cross-machine timestamp comparison sits under cache entries that expire before they are written.
The Engineer · Build desk

What happened
- A dev.to walkthrough traces a reverted profile update to a three-line conflict resolver that kept the write with the larger timestamp while one machine's clock ran about 90 milliseconds ahead of the other.
- The post lists four other symptoms of the same assumption: cache entries expiring before they were written, traces showing responses before requests, rate limiters resetting mid-window, and dedupe windows dropping real events.
- It attributes the reordering to NTP correcting in steps, which after a partition, a VM migration or a long pause can jump a clock backward as well as forward.
- Its central argument is correlation: clocks disagree most during partitions, failovers and overload, the moments when conflict-resolution code runs hardest.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure There is nothing to alert on. The write is accepted, persisted and then discarded by design, so the first report comes from the customer whose change reverted.
- constraint Tighter time sync shrinks the reordering window, and the window is still there. A correctness path that compares wall clocks across machines ends up wrong at a lower rate.
- decision Teams running replicas on cloud instances now have a call to make on opting into the provider's dedicated time service. That changes the odds; the resolver's design is untouched.
- capability Moving every duration and timeout onto a monotonic clock removes the negative-elapsed-time class of bug and leaves conflict resolution as it was; the resolver still needs causal ordering.
That comparison inverts under one condition: the real-time gap between two writes has to be smaller than the disagreement between the two clocks that stamped them [1][2]. At the 90 milliseconds in the post's example, any two updates to the same key that land on different replicas less than 90 ms apart can resolve backwards [1]. Updates further apart resolve correctly. So the exposure tracks how often a single key is written twice in quick succession.
Ninety milliseconds is a small amount of drift to accumulate. The post puts commodity quartz oscillators in the tens of parts per million, where one ppm is one microsecond per second [6]. Ten ppm is 36 ms per hour [2]. A machine at that rate with no correction is 90 ms out in about two and a half hours [3]. On a healthy network the post says NTP holds machines within a few milliseconds of a reference clock [7]. Call that five: the example's skew is roughly eighteen times larger [4]. Corrections arrive as steps, and after a partition, a VM migration or a long pause the step can go backward [8].
The 90 ms is a worked example, and the post does not report how often this happens in production [16]. For it to describe your system, three things have to hold. Replicas stamp writes with their own wall clock. The resolver compares those stamps across replicas. Clients update the same key inside the skew window. According to the post, cloud instances are generally harder to keep in tight sync than bare metal unless you opt into the provider's dedicated time service [9].
Some of the symptoms in the post are subtractions on one machine: a negative t2 minus t1 after a backward step, a rate limiter that resets mid-window [8][5]. The fix for those is a monotonic clock. It only moves forward, and the post recommends time.monotonic(), System.nanoTime() or CLOCK_MONOTONIC for every duration and timeout [11]. That change leaves the conflict resolver where it was, because a monotonic clock's zero point is usually machine boot, so its values are meaningless across machines and often across reboots [12].
What is left is an ordering question, and Leslie Lamport's 1978 paper answered it with causality: an event happened before another if it came first in the same process, if it is the send of a message the other receives, or if a chain of those links them [13]. Where no such chain exists in either direction, the events are concurrent, which the post says "means the system holds no evidence that one came before the other" [14]. A resolver handed two concurrent writes has two honest options: keep both and let the application merge them, or carry a clock that encodes causality. The post names Lamport clocks, vector clocks, and the hybrid clocks it says modern databases actually run on [15].
What to watch
- Whether the post's promised treatment of vector and hybrid clocks quantifies the per-key metadata cost of keeping causal information.
- Any measured skew distribution from a production cluster. That would say what share of writes fall inside the reordering window.
- Evidence that a provider's dedicated time service shrinks the size of post-migration clock steps.