Skip to content

Build1 publisher3 min readPublished

A UK regulator's exit requirement put Form3's payments platform on three clouds at once

Form3 deliberately welded its first payments platform to AWS, then rebuilt it to run active-active-active across three providers after the UK's banking regulator said banks must be able to exit a cloud.

The Engineer · Build desk

Photograph accompanying A UK regulator's exit requirement put Form3's payments platform on three clouds at once
Photo: infoq.com

What happened

  • In 2021 the UK's banking regulator warned that with banks moving to a limited number of cloud providers, an outage at any one of them would take a big chunk of UK financial services with it.
  • One of the largest banks in the UK, a Form3 customer, responded by saying it needed a multi-cloud strategy, and Form3 concluded it had to run its own platform on multiple clouds.
  • Form3's v1 platform ran Java microservices in Docker on AWS ECS, coordinated asynchronous workflows through AWS SQS, and kept its data in a Postgres database on RDS.
  • The v2 design goal was active-active-active, treating each cloud provider like an availability zone so that a whole cloud can fail without callers seeing anything.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost Database backups, multi-AZ placement, the container runtime and the queue move from an AWS invoice onto the platform team's rota, so exit capability is bought with permanent operational headcount.
  • constraint Committing to one build that runs anywhere limits product engineers to what all three providers can support, which is the intersection of their feature sets rather than the best of each.
  • precedent A supervisory expectation aimed at banks reached a supplier's architecture through one customer's requirement, and any vendor holding payment state is reachable by the same route.

The lock-in in v1 was deliberate. "We knowingly coupled ourselves to AWS as tightly as we could because we wanted to offload the responsibility of running technology to a large cloud provider so we wouldn't have to run it ourselves," Kevin Holditch, Form3's VP of engineering, said [7][5]. The team was between four and ten engineers for a one-to-two-year stretch around nine or ten years ago [6]. AWS backed the database up and ran it across availability zones [9].

Three of the four named building blocks in that stack are AWS services: ECS as the container runtime, SQS as the message bus, RDS for Postgres [8][18]. Running on three providers means the Form3 platform team operates a replacement for each of them, itself [13]. Product engineers keep one build of the payments software; Holditch said maintaining one version for AWS, another for Google and another for Azure "would just be a maintenance nightmare" [14].

The property that carries up a level is retry. In v1, a request that failed in one availability zone was retried in another, and Form3 got proficient at that [11]. v2 asks whether a cloud provider can be treated the same way, so that if any cloud fails "the world would carry on running like nothing had happened" [12].

That has a capacity price. If traffic splits evenly across three clouds and any one can drop out with nothing visible to callers, each cloud carries a third of the load while being sized for half, which puts steady-state utilisation at about 67 per cent of provisioned capacity [19]. For the figure to transfer to your system, the split has to be genuinely even and every in-flight request has to be safe to re-drive on another provider. Form3's published transcript gives neither the split nor the failover budget [17].

The talk is titled "How To Run on Three Clouds at Once, and When Not To" [16]. The transcript covers the regulator, the v1 stack and the three v2 design goals, then stops mid-sentence while describing a Kubernetes cluster, which is roughly where the interesting part starts [17]. So the case for not doing it is only legible as the inverse of the work Form3 took back: backups, multi-AZ placement, the container runtime, the queue [9][8].

The trigger was documentary. According to Ross McFarlane, who introduced the talk, the key message of the 2021 regulator statement is that banks must be able to exit a cloud [2][5]. Form3 connects banks and financial institutions to payment schemes in the UK, Europe and the US, so it sits inside its customers' exit plans [4]. "We were successful as we were growing, and those requirements were pushed down onto us," Holditch said [15].

What to watch

  • Publication of the rest of the transcript, which would show the v2 Kubernetes and data replication design that the excerpt cuts off.
  • Whether cloud exit capability starts appearing as a contractual test in bank procurement of payment processors, not only in the banks' own cloud plans.
  • Any Form3 figure for the cost or latency of running the same payments build across AWS, Google and Azure simultaneously.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories