Build1 publisher3 min readPublished Updated
incident.io backs a fail-open rate limiter with a top-k counter in each pod's memory
The on-call platform keeps its rate-limiting counters in managed Valkey under a 99.99 percent SLA, and an intern project adds a local buffer for the busiest gates so that a store outage would not switch limiting off entirely.
The Engineer · Build desk
What happened
- incident.io's Reliability team keeps its rate-limiting counters in Valkey, used through Google Cloud Memorystore, a managed service with a 99.99 percent SLA.
- If that store goes down the limiter fails open, meaning nothing is rate limited at all, a state the company says could lead to system overload.
- An intern on the team shipped a customer-facing notification toggle in his first week and was handed the rate-limiting redundancy problem in his second.
- The design tracks the top-k busiest traffic gates in each pod's local memory and keeps an in-memory buffer for only those gates, so the hardest hitters stay limited while Valkey is down.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A pod-local buffer counts only its own traffic, so the ceiling actually enforced during an outage rises with fleet size: N pods admit up to N times the configured limit.
- decision Picking k decides which limits survive a store outage, because every gate below the cut keeps the old fail-open behaviour.
- cost The gap being closed is roughly 52 minutes a year of unenforced limits, and an engineer's project is the price of closing it; teams whose worst case is cheaper than an overloaded on-call platform will not clear that bar.
A distributed limiter works because every pod consults the same counters. incident.io keeps those counters in Valkey, run as Google Cloud Memorystore [1]. When that store is unreachable the calling code has two choices: reject the request, or let it through. incident.io lets it through. Anthony, the intern who took the project, wrote that with Valkey down the platform would not rate limit at all, and that this could lead to system overload [3].
A 99.99 percent SLA [1] allows 52.56 minutes of downtime a year, which is 0.01 percent of 525,600 minutes [11]. Anthony wrote: "But 99.99% isn't good enough for us - our rate-limiting system is such a critical part of how we keep the platform up that Valkey being down for any amount of time just isn't acceptable." [2]
The fallback he scoped holds two things in each pod's local memory: the top-k busiest traffic gates, meaning the named limits every request is checked against, and an in-memory buffer for just those gates [5]. Two properties follow from where that state sits. Reading it takes no network call, so the fallback does not depend on the system it is covering [15]. Each pod also counts only the requests it handled itself, so with N pods the ceiling actually enforced during an outage is N times the per-pod number [12].
Gates outside the top-k have no local buffer, so they keep failing open while the store is down [13]. The buffer's stated purpose is to rate limit the hardest hitters [5].
Before kickoff Anthony wrote a product scope and a technical scope, iterated with the engineer Johanna and the rest of the team [9]. The post ends with an appendix defining the terms it uses [10]. Most fallback paths get less documentation than that.
Space-saving was the algorithm going in. At kickoff, which the post describes as the meeting where the team stress-tests an approach before work starts [14], HeavyKeeper came up as an alternative, and Patrick, one of the team leads, encouraged benchmarking the options [6]. I would expect a result there to be a statement about incident.io's own distribution of gate traffic: an estimator that names the heavy keys accurately when a handful of gates dominate is not automatically the better pick when the load is spread thin, and the transfer depends on your skew, your k, and how much per-pod memory you will spend on counters. The published account breaks off at that point and reports no numbers for either algorithm [7].
What the design does buy is independence. The Memorystore SLA is a promise about one managed service, and every pod's limiter inherited it [1][3]. After this change, the worst case during a Valkey outage is partial enforcement at a looser aggregate ceiling [12][13].
What to watch
- Whether incident.io publishes the Space-saving versus HeavyKeeper benchmark, and the per-pod memory each costs at its chosen k.
- Whether the fallback leaves the tail of gates unlimited or assigns them a fixed default allowance during an outage.
- Whether a real Memorystore outage exercises the local buffer, and what the enforced aggregate ceiling turns out to be.