Build1 publisher3 min readPublished
Two pay requests read the same order as pending six milliseconds apart and both charged it
A dev.to post-mortem's request log puts 89 milliseconds between the guard read and the paid write, and the second request's retry landed its charge more than four seconds after the row already said paid.
The Engineer · Build desk
What happened
- A support queue took six tickets in 48 hours from customers who had been charged twice for a single order, and engineering had not found the cause in the code.
- The database showed one order, ord_20394 for $87.50, with two distinct payment records recorded four seconds apart, both of which resolved successfully.
- In the payment method, the only thing standing in front of the provider call is a conditional on the order's status column, which throws a conflict unless the status reads pending.
- The HTTP log shows both requests reading the row as pending before either had written anything, r1 at 14:22:27.421 and r2 six milliseconds later at 14:22:27.427.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A status comparison in application code cannot order two callers, because the read it depends on is a snapshot. Making the pending-to-paid transition exclusive means moving the guard into the write.
- decision Engineers have to pick the layer. A conditional update stops a second caller before it charges; an idempotency key is what covers a retry whose first attempt's fate the caller cannot determine.
- cost Every duplicate that reaches a customer is paid for twice over, once as a manual refund and once as support handling, before anyone knows which code path produced it.
The window is 89 milliseconds wide. Request r1 read the order as pending at 14:22:27.421 and wrote paid at 14:22:27.510 [8][10][1]. Anything that read the row inside those 89 milliseconds saw pending and was cleared to charge. r2 read at 14:22:27.427, six milliseconds after r1 [8][2]. The post's own trace calls that gap eight milliseconds, which is the distance between the two POST lines at .410 and .418 [15][7][3].
The guard is one line: `if (order.status !== 'pending') throw new ConflictException(...)` [5]. It compares a field on an object that `findById` already copied into the process. `findById` issues a plain SELECT, which the post describes as a snapshot read: it returns the row's value at the time of read, does not guarantee that value later, and does not stop other transactions from reading or writing [6]. The write at the end of the method is unconditional, `updateStatus(order.id, 'paid')` [16]. So r2's write is a no-op overwrite, because nothing enforces that the row was still pending at write time [14]. The post calls this "a classic case of a check-then-act race condition" [13].
r2's first charge attempt aborted with ECONNABORTED at 14:22:29.431, two seconds after the call, which is what a two-second client timeout looks like [9][11][5]. The retry succeeded at 14:22:31.802, 4.292 seconds after the orders row already said paid [12][6]. No application code runs between those two attempts; the retry loop lives inside the payment client, below the guard. Both requests returned 200 OK, r1 in 102 milliseconds and r2 in 4394 milliseconds [10][12]. From the client's side, nothing failed.
Two candidate fixes sit at different layers, and this log picks one of them. r2 called charge at .431, 79 milliseconds before r1 committed paid [9][10][9]. A read that took a row lock, or a conditional `UPDATE ... WHERE status = 'pending'` that moved the order out of pending before charging, would have blocked or failed r2 at that point, and the second charge would never have been requested. Serializing at the row is sufficient for this incident. It does not cover the case r2's retry raises, where a charge attempt aborts and the caller cannot tell whether the provider booked it. In my view that case needs an idempotency key on the charge call, because the deduplication then happens on the provider's side of the connection. The post breaks off in its fix section [17].
For the timings to transfer to your service, two things have to hold. The endpoint has to be reachable twice in quick succession, which it was here: two POSTs to /orders/ord_20394/pay, eight milliseconds apart [7][3]. And the exposed window is the read-to-write distance, which is dominated by the provider round trip. r1's charge took 73 milliseconds of a 102 millisecond request [4][10]. A provider that answers in two seconds hands every concurrent caller a two-second window in which the same check passes.
The two charges on ord_20394 came to $175.00 [7]. That refund, and five others, went out by hand while the cause was still open [1][2].
What to watch
- Whether the payment provider's charge API accepts an idempotency key, and how long it deduplicates on one, since that decides whether the retry path can be closed at all.
- Whether the fix turns the pending-to-paid transition into a conditional update, and what the endpoint then returns to the losing caller.
- Whether six tickets in 48 hours is the duplicate-charge rate or only the customers who spotted a second $87.50.