Build1 distinct publisher3 min readPublished
Metering bytes processed made a heavy scan look like a cheap point read, so Uber scrapped central quota counting in Redis and pushed overload decisions down to the storage layer, next to the partitions that actually saturate.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
Price a read in bytes processed and you are metering the wrong end of the pipe. A query that scans a whole table and returns one row was charged the same capacity cost as a query that reads one row, which Uber attributes to the way MySQL handles scanning and filtering [12]. The bytes visible at the metering point are the bytes that come back. The work is the scan. The error in that price therefore grows with the size of the table, and the tenant running the worst query is the one billed least.
The second defect is placement. Data is sharded across partitions, each one leader plus two followers coordinated via Raft [6], so three nodes per partition [18]. The partition is what runs out of headroom, not the cluster. For a stateless routing layer to shed on behalf of an overloaded partition, it would have to hold realtime health and load for thousands of partitions, and Uber says that tracking overhead undermined the scalability of the architecture [11].
The accounting had its own bill. Quota usage sat in a central Redis cache because the routing nodes keep no state [9], and every request paid a call to it [10]. At the aggregate rate the same post quotes, tens of millions of requests per second [3], that is a counter store fielding operations in the same order of magnitude as the database it protects [19]. Putting Redis in front of every request to improve availability is a hard sell at review.
A static quota is one number per tenant, fixed when someone asked for it. Uber reports the predictable result: frequent requests from stakeholders to adjust quotas, and quotas that were ineffective in multitenant environments [13]. The requirement it was meant to satisfy is fairness under a mix that moves, with workloads differing in traffic shape, latency profile and system impact [16]. One load manager here covers a transactional CRUD store and an append-only one [20]. A constant cannot express shape. It trips at the same count whether the burst clears in a second or a scan grinds for a minute, and it is the sustained case that turns into downstream timeouts and piled-up retries [17].
Whether the fix transfers depends on properties of your system rather than Uber's. Your storage nodes have to expose load locally and cheaply, or you have rebuilt the tracking problem one layer down. Your callers have to retry, because rejecting early only beats queueing if someone comes back. And your tenant count has to be high enough that hand-editing limits is a standing tax [13]. With three tenants and an engine that reports rows examined, a static quota is boring and debuggable, and boring is worth a lot at 3am.
What I cannot check is the signal. Uber says the load manager detects overload from multiple signals [15], and the next section of the post is titled Identifying the Right Signal for Overload [22], which is where the copy I have stops. Adaptive admission control is worth exactly what its signal is worth: choose one that lags and you shed after the damage; choose one that is noisy and you shed healthy traffic. The narrow, testable claim from the rest of it stands on its own. Admission control decided at a layer that cannot observe rows examined is pricing a proxy, and nothing bounds that proxy's error [12].
Ranked by verification strength, evidence, and original report placement.
Uber's thousands of microservices handle traffic for over 170 million monthly active users, including riders, Uber Eats users, drivers and couriers.
Docstore and Schemaless are Uber's in-house distributed databases built on top of MySQL.
Docstore and Schemaless span thousands of clusters, store tens of petabytes of operational data, and serve tens of millions of requests per second with billions of rows read or updated.
Both Docstore and Schemaless comprise three primary layers: a stateless query engine, a stateful storage engine, and a control plane.
The stateless query engine handles query planning, request routing, sharding, schema management, authorization, request parsing and validation, coordinating and validating client requests before handing them to the storage layer.
The stateful storage engine handles transaction management, connection pooling, consensus and replication; data is sharded across partitions, with each partition consisting of one leader and two followers coordinated via Raft for strong consistency.
Distinct publishers with included, body-backed reporting in this cluster.
Follow any of these and your For You feed starts watching them — no settings page required.
build
The database was fine: a one-in-three DNS failure hidden by an app that logs nothing1 distinct publisher
build
1,254 dead mutants, a 100% score, and a payment charged twice1 distinct publisher
build
Redlock's five masters are the price of a lock that survives failover1 distinct publisher
invest
A bipartisan push against Flock puts 6,000 municipal renewals in play1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
First-hand, unchecked
Everything here comes from the people who run the system, and only from them. That makes the architecture facts about as good as such facts get — partition shape, hardware, the precise metering rule — and it makes the one forward-looking assertion, that the replacement keeps things stable and fair, unfalsifiable as written. No latency curve, no shed rate, no incident count accompanies the new design.
Real load, one fleet
The load is unquestionably real by Uber's account — thousands of clusters, tens of millions of requests a second — but the adoption is one company's internal fleet, self-reported, with no library, no release and no second operator confirming the approach. What travels beyond Uber is the lesson about byte-based metering; the load manager itself does not leave the building.
Victory lap over a postmortem
"Conquered" is carrying weight the body never lifts. Most of the post is a candid account of Uber's own failure, including a metering rule it calls a fundamental flaw — that honesty pulls the gap close to zero. What pushes it back positive is the unmeasured triumph on either side of it: an intelligent load manager announced in the opening and still unquantified by the point the account stops.
Brand, not billing
Nothing is for sale. No product, no license, no round — what Uber earns from this is recruiting signal and standing among infrastructure engineers. That reward structure favours a clean three-act arc (naive attempt, hard lesson, intelligent solution) and quietly discourages publishing the parts that spoil the shape, which is roughly where the missing numbers would sit.
Single witness
We hold the mechanism firmly and the outcome loosely. For how Docstore and Schemaless are put together, the only witness is also the best possible one; for whether pushing enforcement down to storage worked, there is no one else in the room. Two of the sharper points in our coverage are arithmetic on Uber's figures rather than Uber's statements, which is safe but is still inference — and one independent measurement would move this number in either direction.