Build1 publisher2 min readPublished
Canva credits a two-week-old backpressure rollout with absorbing a cloud outage
Canva says the worker slowed itself as its dependencies failed and sped back up unattended. The case the post builds is against circuit breakers, whose only settings are full speed and full stop, and against static rate limits.
The Engineer · Build desk
What happened
- Canva began rolling out Worker Backpressure earlier this year, and roughly two weeks in, a major cloud-provider outage sent error spikes across many of its services, including a critical queue worker.
- Canva says that worker slowed itself as its dependencies failed and sped back up on its own once they recovered, with the dead letter queue quiet and nobody paged.
- The default it replaces is a greedy worker that grabs a message the moment one lands and it has spare capacity. That is what gives minimum latency while everything downstream is healthy.
- Canva runs thousands of queues with diverse business logic and dependencies. Any single throttling policy has to cover that whole fleet.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The bill being removed here is on-call time, not compute: draining and reprocessing a dead letter queue is manual work in many cases, and it is paid by whoever holds the pager while the dependency recovers.
- constraint A per-worker loop regulates only that worker's pull rate, so a shared datastore can still be pushed over by clients Canva is not throttling.
- decision Adding workers during an incident becomes the step to question first, since Canva's position is that scaling up can unleash more load on the dependency that is already failing.
A greedy consumer and a slow datastore make a specific shape of incident. A user clicks Export, a message lands on a queue, and a worker fetches design data from a database before calling a rendering service [5]. If that database is already slow, perhaps under a background migration, the worker keeps pulling at full speed [5]. Canva's writeup says a few slow responses from the database can then snowball into a high-severity incident with exports failing for thousands of users [6].
The most useful part of the post is the case against circuit breakers. A breaker counts errors, trips open when a threshold is crossed, and stops all traffic until a cooldown expires [8]. It has two settings. Canva's objection is that there is no gradual ramp between full speed and full stop. The sudden flood of resumed traffic can knock over a dependency that had only just recovered [9]. Exponential backoff has a problem of scope instead: it smooths individual retry storms, but it works per message and never regulates the overall rate at which a worker calls a dependency [10].
Static rate limits fail on maintenance. A fixed limit is only correct for a fixed world, the post argues, and capacity changes constantly, especially for shared dependencies, so a stale limit either throttles for nothing or sits far above real capacity [12]. At the fleet size Canva describes, one hand-chosen limit per queue means thousands of numbers to keep current [13]. "Finding one answer that works across our diverse fleet is far from trivial," Canva's engineers wrote [14].
The post is labelled Part 1 [20]. It puts the control questions in its own words: should the worker stop entirely or just slow down, by how much and for how long, and what signals should drive that decision [18].
The outage is the only evidence the post offers for the new default. Canva does not name the provider, and it dates the rollout as earlier this year [17]. Two conditions decide whether the result travels to another fleet. The work has to be deferrable in the way Canva describes, where a briefly failing dependency still leaves the user's request successful and the job waiting on the queue [15]. And the consumers you can throttle have to be a large share of the load on the thing that is failing. A worker governs only its own pull rate [4], and the dependency is often a shared resource such as a datastore [16].
What to watch
- Part 2 publishing the actual signals and thresholds would show whether the loop is driven by error rate, latency, or both.