Build1 publisher2 min readPublished
Cloudflare reclaimed more than 100TB of RAM by shrinking one consistent-hashing structure
Cloudflare's excess memory sat in pingora-ketama, the open-source consistent hashing library inside its internal load balancer. An engineer named Ivan filed the ticket that started the work.
The Engineer · Build desk
What happened
- Cloudflare says small changes to a single algorithm cut the memory footprint of one of its Pingora-based services enough to reclaim more than 100TB of RAM globally.
- The finding was that Pingora Backend Router, its internal load-balancing service, was using significantly more memory than expected in structures associated with pingora-ketama.
- The saving came on top of 100TB of memory that Cloudflare's DNS team shed the previous month.
- Cloudflare routes cacheable requests to servers by URL using consistent hashing, so each data center stores one copy of a file and has a stable way to find it.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Headroom at this fleet size is now something an engineer can file a ticket for. The thing that returned 100TB was a change to a hashing structure inside a dependency.
- exposure pingora-ketama is published as open source, so anyone who built routing on it is holding the same structures in their own load balancer and can measure the same resident set.
- cost The saving is denominated in RAM. It converts into capacity only where memory was the limit on how many services a node could run, and Cloudflare's stated requirement is that every service runs on every node.
- precedent Cloudflare gives cross-team resource use a named owner in its Performance team. That makes another team's allocations somebody's standing job.
Cloudflare's post lays consistent hashing out on a number line instead of the usual ring. Servers are placed by the hash of a value like an IP address, tasks by the hash of a cache key, and each task is assigned to the first server to its left [11][12]. One server's range wraps from the maximum value back to zero, which is where the ring picture comes from [13]. A server's share of requests is proportional to the width of its range [14], and in the post's own three-server example, server A's range is significantly larger than B's or C's [15].
Hash outputs are essentially random numbers, so those widths have to be treated as a distribution, with an expected value and a standard deviation [16].
A few megabytes per process matters here because of how the fleet is deployed. Cloudflare says it runs thousands of servers worldwide with petabytes of RAM and millions of CPU cores, all pushed to the max [8]. "As vast as those resources feel, they are still finite, and when you need every service to run on every node, it doesn't leave room for wasted space," the post said [9]. It also says small improvements are greatly magnified at that scale, so even 1%-at-a-time improvements are worth celebrating [17]. Spread across 10,000 machines, more than 100TB works out to more than 10GB each [19].
Cloudflare credits its Performance team with keeping resource sharing between teams balanced [7], and the story starts with a ticket from an engineer named Ivan, titled "Excessive memory usage from pingora-ketama in Pingora Backend Router" [3]. pingora-ketama is Cloudflare's open-source library for consistent hashing [5], and the service in the title is the internal load balancer [4]. The post concedes its own acronym: "yes, PBR" [21].
The claim that an algorithm review is cheaper than a capacity order needs numbers, and the post breaks off in the middle of the probability lesson, before the change made inside pingora-ketama and before any comparison of engineering time with the price of RAM [20].
What to watch
- The remainder of the post: which change to pingora-ketama produced the saving, and whether it lands in the public crate.
- Whether the freed memory is reallocated to other services or shows up as higher service density per node.
- Whether the DNS team's 100TB last month came from the same class of data structure as this one.