Leadership1 publisher3 min readPublished
A Meta infrastructure lead argues that who degrades first belongs in writing before the outage
Manas Chaudhari, tech lead for WhatsApp's messaging infrastructure at Meta, says forecasting capacity is the easy half, and the hard half is agreeing in advance which customers and features get dropped first.
The Board Room · Leadership desk
What happened
- Manas Chaudhari, tech lead for WhatsApp's messaging infrastructure at Meta, wrote in Forbes that capacity planning is largely automated and that most organizations do it well. It is almost never the cause of the outage.
- He cites researchers who studied 21 major outages across 11 organizations and found that in more than half the escape route was shedding load: throttling requests or turning traffic off.
- Once capacity runs short, he wrote, the infrastructure picks who gets less on its own. It knows nothing about which customers matter, and it favors whatever is loudest.
- He names two levers that already exist to buy, isolation and shedding, and says what is missing is the input they need: a pre-agreed order of expendability.
Compiled by The Board RoomSomething wrong?How this is made
Why it matters
- decision The ranking of customers and features gets made in every incident whether or not anyone writes it down, so declining to write it hands a commercial judgement to a queue discipline.
- cost Both remedies bill someone: isolation is paid for in idle capacity, shedding in the revenue of the requests dropped. The trade lands in front of finance.
- exposure The tenant that absorbs the default degradation is the smallest one sharing a shard with the largest, and it learns its position from an outage instead of a contract.
- constraint Because the same failure is called noisy neighbors, hot keys, head-of-line blocking or just slow depending on who is describing it, it seldom arrives at the level where the ranking could be set.
The reason the default allocation is bad has to do with retries. When a component fails it retries, and retrying generates more load, so the component in the worst shape is the one asking for the most. Chaudhari wrote that capacity therefore ends up allocated "in a roughly inverse proportion to health, and health has nothing to do with importance" [12]. He put the comparison bluntly: "This is worse than a coin toss, which would at least be fair." [13] A background export nobody would miss can retry its way into the connection pool that checkout depends on, and win [14].
The healthy case is the one that matters more for planning, because nothing has malfunctioned. A feature goes viral, takes most of the requests, and so takes most of the capacity. The critical path carrying a fraction of the volume and most of the revenue gets what is left, because share of traffic was the only signal the system had [15].
A better forecast does not close this gap, and the numbers on a spike show why. Chaudhari wrote that "A goal in a World Cup final moves a large part of the planet to their phones in the same second" [9], with load jumping to two or three times the usual peak and fanout multiplying each request on arrival [10]. Provisioning to absorb that means carrying 100 to 200 percent headroom above peak [11]. Isolation, the other lever he names, is also paid for in utilization [18].
I would expect the objection to a written expendability order to be a commercial one. It is a document, and a large customer's procurement team can ask to see which tier it sits in. The answer is that the order already exists. Today it is set by a partition boundary, a queue discipline or whichever connection pool empties first, and Chaudhari wrote that it usually falls on the smallest customer, the one sharing a shard with the largest [17].
The strongest number in the piece is also the thinnest sourced. The finding that more than half of 21 major outages across 11 organizations ended in shedding load is attributed to "Researchers", and the article does not name the study [21]. That makes at least 11 of those 21 [5] an indication of how often teams reach for throttling under pressure, not a measured base rate.
None of this is a this-week problem. Ranking customers, tiers and features in order of expendability is a commercial judgement that has to survive a sales team and a contract, so it takes a quarter, not a sprint. Both levers are available to buy and neither is exotic [18]; what they need before they are worth anything is the ranking [19].
What to watch
- Whether the 21-outage study gets named, so the more-than-half finding can be checked against a defined population of incidents.
- Whether Meta's published practice for product engineers gets copied by multi-tenant vendors as internal policy.
- Whether any vendor writes an expendability tier into an SLA, putting the degradation order in front of customers before an incident.