Skip to content

Build1 publisher2 min readPublished

A paid M-Pesa ticket went undelivered because fulfilment threw after the SUCCESS write

A reconciliation job verified the charge with Paystack and marked the payment row SUCCESS, then fulfilment failed on an unrelated tenant-context bug, and the sweeper only ever reads PENDING rows.

The Engineer · Build desk

Illustration accompanying A paid M-Pesa ticket went undelivered because fulfilment threw after the SUCCESS write

What happened

  • A customer completed an M-Pesa STK Push prompt with their PIN, saw no confirmation in the app, and the payment record stayed on PENDING.
  • Logging added at charge initiation returned http=200, status=true and dataStatus=pay_offline, so the charge had gone out and the confirming webhook was the part that never got processed.
  • A scheduled job that sweeps PENDING payments re-verified the charge with Paystack, confirmed real money had moved, set the row to SUCCESS, and the customer still received no ticket.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint The recovery set is whatever the sweeper's WHERE clause matches, so any terminal status the job writes is beyond its own reach on every later run.
  • decision Ordering fulfilment before the status write is the choice on the table, and it moves the risk onto a step that may execute twice instead of onto a customer who gets nothing.
  • exposure Alerting built on stuck payments will not see a settled-but-undelivered row, which leaves the paying customer as the detector of last resort.
  • cost A payment path with no log lines costs a full incident just to become observable before any fix can be designed.

The line that decides which payments can ever be repaired is the query predicate: `findMany({ where: { status: 'PENDING' } })` [8]. Status on that row records what Paystack said about the charge, and delivery of the thing bought happens in a separate step [11]. So the job's write removes a row from the job's own read set. Once it sets SUCCESS, that row can never match PENDING again [15].

Justin Wilson, commenting on the author's earlier post about STK Push timeouts and webhook verification [1], said: "The key missing safeguard is idempotency. Store transaction keys and make retries safe so delayed callbacks cannot create duplicate orders." [2] The author added exactly that. Ticket issuance is guarded by an already-issued check for the payment, and stream access is an idempotent upsert keyed on user plus resource, so two runs of the job, or three replicas running it at once, cannot produce two tickets for one payment [9].

Verification worked too. The job found the stuck payment, confirmed with Paystack that real money had moved, and flipped the row to SUCCESS [10]. Fulfilment then threw partway through, on a tenant-context bug in how the job ran that had nothing to do with payments [12]. By then the status write had landed [12].

From that moment the payment was outside every recovery path in the system. The sweeper reads PENDING [8], and this row said SUCCESS, so nothing looked at it again and nothing was watching for the gap [13]. The customer had paid, and the system treated the transaction as closed [13].

The author's fix is to invert the order and fulfil first [14]. That makes fulfilment the step that can run more than once, and the guards already cover that case: the already-issued check and the user-plus-resource upsert both absorb a repeat [16]. For the inversion to transfer to another codebase, fulfilment has to be idempotent under a key that holds across replicas. Where it is not, the failure mode changes from no ticket to two.

The other option is to keep the order and add a second sweep over payments that are settled but undelivered, which needs a field to sweep on. The post describes status values on the payment row and no separate fulfilment flag [17].

Either fix depends on being able to see the path. Before logging was added at charge initiation, the payment code emitted no log lines at all [4], and the first thing the new line printed was `http=200 status=true dataStatus=pay_offline` [5], which is Paystack's state for waiting on the customer to finish on the phone [6].

What to watch

  • Whether the author publishes the reordered job, and whether the write order holds when fulfilment fails on a second attempt.
  • Whether the tenant-context bug that broke fulfilment is fixed in its own right or only tolerated by retries.
  • How often Paystack's confirming webhook drops, which sets how much of delivery rests on the sweeper.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories