Skip to content

Build1 publisher3 min readPublished

Amex's cell boundary keeps a failing payment inside the cell that started it

Ben Cane's account of American Express's cell architecture shows that the state a second cell would need to take over a mid-flight transaction is exactly the cross-cell dependency the design bans, so isolation stands in for failover.

The Engineer · Build desk

Photograph accompanying Amex's cell boundary keeps a failing payment inside the cell that started it
Photo: blog.bytebytego.com

What happened

  • American Express describes its design requirement as approval or decline almost instantly, with the entire processing of a payment finishing inside that short window.
  • The platform began moving onto cloud-native infrastructure in 2018, which forced the team to drop the older assumption that hardware is engineered to stay running.
  • The cell is defined by five properties, ending with a ban on synchronous cross-cell dependencies anywhere in the critical path.
  • In the scenario Cane walks through, a service inside the processing unit begins failing partway through the chain while the customer at the checkout terminal is still waiting.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost Every cell owns its own databases and supporting infrastructure and every cell needs the replicated reference data, so capacity and licensing multiply by cell count; the account puts no figure on either.
  • capability Because a cell can be pulled out of rotation while the rest of the platform keeps processing, maintenance and incident containment stop requiring a quiet window in the traffic.
  • constraint Any future feature that needs a synchronous global view inside the payment path is off the roadmap for as long as the fifth property holds, which makes the boundary a standing veto over product design.
  • decision Teams still running on hardware engineered to stay up have to decide whether they accept the vanishing-server premise, because the duplication only pays back under that assumption.

Property five is the load-bearing one. A cell has no synchronous cross-cell dependencies in the critical path [13]. Set that beside the failure case: a transaction is partway through the chain, a service under it is failing, and moving the work to another server would require the second unit to know how far the first one got [4]. That knowledge is a synchronous read across a cell boundary, inside the critical path, so the definition of a cell rules out the one mechanism that would rescue the request [15]. American Express is direct about why: sharing that progress state is what turns two independent units into one fragile system [5].

Redundancy is still present in the ordinary sense. Transactions are routed into one of several independent processing units [3], and each unit is a complete copy of the stack, with its own microservices, databases, DNS and supporting infrastructure inside the boundary [12]. Those copies are where the next transaction goes. The transaction already inside the failing cell has nowhere to be handed to, because the handover is itself the coupling [5].

Cane's team reports that most of the engineering effort goes into keeping that isolation intact under pressure [6]. That is the sentence to take seriously. A cell boundary is not a one-time build; it degrades the first time someone adds a convenient synchronous call across it because the deadline was closer than the review.

Both rejected patterns trace back to the clock rather than to volume. Event-driven processing fit poorly because the answer itself still has to arrive fast, even though asynchronous work exists elsewhere in the flow [9]. A monolith was rejected on scaling [10]. Neither rejection comes with a figure. The account describes the requirement as approval or decline almost instantly and never converts that into milliseconds [1][16], so there is no published budget to hold your own p99 against, and no cell count to divide traffic by.

Transfer needs a workload where each request completes inside one cell and where something upstream owns the retry. The first condition follows from the properties as stated [13]. The second is unaddressed. American Express sits in the middle of the chain, taking the transaction from the acquiring bank and delivering it to the card issuer that holds the account balance [7], which makes the acquirer the natural place for a failed authorisation to be reissued, but the published account does not say who retries or what the customer at the terminal ends up seeing [18].

Cane dates the thinking to the SOA era, when his teams were already trying to limit the blast radius of a single failure, before microservices existed as a concept and before anyone had coined the term cell-based architecture [11]. The naming arrived last, which is the usual order.

The isolation properties here are specified tightly enough to test against a running system: independent deploy, owned data stores, single failure domain, drainable, no synchronous cross-cell call in the path [13]. The recovery path for a transaction already inside the failing cell is not described [18], and that is the part a payments team copying this shape would have to design itself.

What to watch

  • Whether American Express publishes a cell count or a millisecond budget, which is what a reader would need to test the shape against their own p99.
  • Whether the no-synchronous-cross-cell rule survives contact with observability and reference data replication, which the published account starts describing and cuts off mid-sentence.
  • Any statement of who owns the retry when a cell fails mid-authorisation, the acquirer or the platform.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories