Build1 publisher3 min readPublished
Saga rollbacks break in the gap between semantic and atomic undo
In a five-step saga, running compensations out of LIFO order leaves a payment captured against a cancelled order, a dev.to walkthrough shows. Its fix is a durable, versioned coordinator with idempotency keys on every step and every compensation.
The Engineer · Build desk
What happened
- Sagas swap two-phase commit's atomicity for a chain of local transactions, each paired with a compensating transaction that undoes its effect in business terms.
- In the post's five-step order saga, a failure at T4 must trigger C3, C2 and C1 in that order, compensating only the steps that committed.
- The coordinator is a durable state machine that persists each saga before invoking a step and persists each commit before moving on.
- A version field on the saga record, checked in a MongoDB conditional update, stops two recovering workers from running the same compensation.
- Each step and compensation carries an idempotency key built from saga ID, step index and phase, and downstream services store it with the result.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Rollback can only reach a business-equivalent state, so any step with an outside side effect, such as an email, needs its own designed follow-up action.
- exposure Compensating out of order opens a window in which a customer is charged for an order the system has already cancelled.
- decision A failed compensation lands in a terminal state, so teams adopting the pattern must decide who owns a saga that can neither finish nor undo.
The dev.to post builds its code around one crash window. If the coordinator dies after T3 succeeds but before it records that success, recovery cannot safely decide whether to run C3 [8]. The answer is write ordering. The coordinator persists the SagaInstance before invoking each step. It also persists the move to StepCommitted before starting the next one [9]. Steps run one at a time, so a crash leaves at most one step per saga with an unknown outcome, sitting in StepExecuting or StepCompensating [14].
Retries settle that step. The coordinator retries on transient failures, including a step that finished but was never recorded, so every step and every compensation must be idempotent [12]. The key is `fmt.Sprintf("%s:step%d:%s", sagaID, stepIndex, phase)` [13]. The downstream service stores it with its result. On re-delivery it returns the cached result without repeating side effects [13]. Putting the phase in the key is a good call. A step's forward call and its compensation get different keys, so a retried refund cannot be answered with the cached charge [15].
The cost moves to the participants. Under two-phase commit, participants hold locks across network round-trips, and one participant going down blocks the whole cohort [1]. The saga drops that cross-service lock contention [2]. In exchange, every downstream service has to accept a client-supplied key and keep it with its result. A payment or mail service that cannot do that cannot meet the idempotency requirement the retries impose [16].
The concurrency guard is the config line I would copy. The update filters on `bson.M{"_id": saga.ID, "version": saga.Version}` and bumps the version with `$inc` [11]. A MatchedCount of zero returns ErrConcurrentModification, and the losing instance re-fetches and re-evaluates [11]. According to the post, this prevents split-brain compensation runs when competing workers recover the same saga after a pod restart [11].
Ordering protects the business invariant. The post requires compensation to be strictly LIFO over successfully committed steps. It warns that nothing reverses execution order by default [6]. In its five-step example, T4 fails and the coordinator runs C3, C2, C1. C4 never runs, because T4 never committed [6][7]. T2 reserved inventory and T3 charged payment. Run C1 first and the order record is cancelled while payment stays captured against it until the other compensations catch up [7].
None of this helps once a side effect has left the system. A compensation is a new forward operation. It reaches a state equivalent to the old one in business terms [3]. If T3 sent an email, C3 cannot unsend it [4]. The customer's inbox was never part of the transaction. C3 can send a follow-up, and the system is then in a different observable state [4].
Compensations also fail. Each one runs over the network against a service that may be unavailable. The post calls a failed compensation a strictly harder problem than the original failure [5]. Its state machine gives that case a status, StepFailed, commented as terminal [10]. The post does not describe what process takes over a saga that lands there [17].
What to watch
- A defined owner for StepFailed sagas, such as an operator queue or a reconciliation job, would close the one failure path the design leaves open.
- Whether each participant, especially payment and mail services, can store a client-supplied key with its result; the retry design fails at any step that cannot.