Skip to content

Build1 publisher3 min readPublished

Expired Redis leases let two Go workers process the same invoice at once

Redis leases in a dev.to Go lock guide expire after 10 seconds mid-job, so a second worker takes the invoice while the first keeps running. The owner-checked release protects only the lock key, so refusing the stale worker's writes falls to the store.

The Engineer · Build desk

Illustration accompanying Expired Redis leases let two Go workers process the same invoice at once

What happened

  • The guide builds on SET resource_name random_token NX PX 30000: create the key only if absent, expire it after 30 seconds, and tag it with the owner's token.
  • In its failure timeline, Worker A's 10-second lease expires mid-job, Worker B acquires the lock, and A then finishes and deletes B's key.
  • The fix it prescribes is an atomic check-and-delete on the owner token, and it rules out a GET followed by a separate DEL.
  • It warns that Redis can apply a SET whose reply is lost, so an acquisition error does not prove that no lock was created.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint An owner-checked release guards the lock key only; a worker whose lease lapsed keeps writing to the invoice until it reaches the release step.
  • exposure Any store that accepts writes because the caller holds the Redis lock is open to a stale second writer, and random tokens cannot tell it which writer is newer.
  • decision Setting the TTL means estimating worst-case job time: a long lease strands a crashed worker's invoice longer, a short one lets slow jobs overlap the next holder.
  • cost Retrying an ambiguous SET with a fresh token can leave the invoice blocked behind an orphaned lease for the full TTL.

From the moment Worker B acquires the lock until Worker A reaches its DEL, both workers are processing invoice 123 [8]. The atomic compare-and-delete the guide prescribes fixes the final step, where A would remove a key it no longer owns [6]. The overlap before that step is unchanged [8].

The usage code shows why. It takes the lock with a uuid token, defers a release under a 2-second timeout, and returns processInvoice(ctx) [9]. Nothing on that path goes back to Redis while the invoice is being processed. The writes to the invoice never pass through Redis at all [12]. According to the post, this is the single-instance pattern Redis itself documents: an atomic SET with NX, an expiry and a random owner value [1]. A sync.Mutex holds because every goroutine shares one process's memory. The Redis workers share only a remote key, with process crashes, packet delay, failover and lease expiry sitting between them [11].

The guide handles this limit carefully in one place. When an acquire returns an error, Redis may still have applied the SET. The post says to generate the token before the command, keep it, and not retry with a new one, since a fresh token can leave the first lease held until it expires [4]. On recovery, it warns, an owner check must account for the remaining lease time, because a matching GET "alone cannot guarantee ownership after the check" [10]. Worker A's writes are in the same position. Whatever A last learned about its lease can be out of date by the time a write lands [12].

That leaves the resource as the only component positioned to refuse a stale writer. The post lists the question among its hard cases: "Can the system prevent a stale worker from modifying the resource?" [7] The available text of the post ends at the release code, before it gets to an answer. The ownership token as written cannot supply one. Random values like "aaa" and "bbb" identify holders but have no order. A store comparing them cannot tell which acquisition is newer. To reject Worker A, the store needs a value that increases with each acquisition, checked on every write [13].

The TTL has to be chosen from both ends. The post makes the lease the recovery path when release cannot complete [9], so a long lease keeps a crashed worker's invoice locked longer. A short lease raises the odds that a live job outruns it, as the 10-second lease does in the timeline [5]. The 30-second PX value in the primitive avoids overlap only for jobs that reliably finish, pauses included, well inside 30 seconds [14].

What to watch

  • Whether the rest of the post adds a resource-side check, such as an increasing token the invoice store compares on every write.
  • Whether Redis failover and clock differences, both on the post's failure list, get treated as separate ways a worker can lose exclusion without knowing it.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories