Leadership1 publisher3 min readPublished
Aerospike's CTO would judge a database at 3x load with a node down and two years of data growth
Srini Srinivasan argues that evaluations run at comfortable utilization only measure the flat part of the queueing curve. His alternative asks for p99.9 under conditions a two-week bake-off has to manufacture.
The Board Room · Leadership desk
What happened
- Srini Srinivasan, founder and CTO at Aerospike, writes that most database evaluations measure speed, capacity, uptime and scalability under favorable conditions, while production brings bursts, data growth and node failures.
- His test of predictability is the same p99.9 response time under three times expected load, while a node is down, and after two years of data growth.
- Queueing behavior is nonlinear, so as utilization rises the curve steepens and response times that were single-digit milliseconds become hundreds.
- Agentic systems decide at runtime how much context to retrieve and how many steps to take, so one request may resolve in a single hop while the next runs 40 sequential retrievals.
Compiled by The Board RoomSomething wrong?How this is made
Why it matters
- cost Declining to measure variance moves its cost onto caches, read replicas and spare headroom. The platform budget carries that for as long as the database stays in place.
- constraint A criterion with three simultaneous conditions cannot be checked on today's dataset at today's load. Buyers have to generate projected volume and inject the node failure themselves, so the evaluation takes longer and costs more.
- decision The choice in an RFP is between a headline throughput figure and a tail percentile measured under a declared fault, and only the second one can be checked at signing.
- precedent A supplier that competes on steady tail latency is supplying the buyer's test language. Measured figures appear for no product in the post, so nothing in it shows who passes the criterion.
The criterion comes from a seller with a stake in it. Srinivasan is founder and CTO at Aerospike, which sells a real-time NoSQL database for mission-critical workloads [1]. He gives no measured latency figures for any named database, his own included [19]. The three conditions he names are the ones a low-variance store is built to survive. The test can still be right, and a buyer should price it before writing it into a document.
The fan-out figures come out of one expression. With a slow-call rate p and n calls in an interaction, the chance of at least one slow call is 1 - (1 - p)^n. At a rate of one in 100 and 100 calls, that is 1 - 0.99^100, or 63.4 percent [16], which is the roughly two in three interactions Srinivasan cites [7]. Drop the rate to one in 1,000 and it falls to about 9.5 percent; one in 10,000 gives about 1 percent [17]. Jeff Dean and Luiz Barroso set out the effect in "The Tail at Scale" in 2013 [8].
That expression treats each call's slowness as independent of the others. Real slowness often has a common cause, and Srinivasan names one: market activity flooding the same small set of keys [15]. Where the cause is shared, the same one-percent tail rate lands in about one percent of interactions instead of 63 percent, and every call in those interactions is slow [18].
Running his test costs more than running a benchmark, because three conditions have to hold at once [3]. Two years of data growth means generating the volume you project for two years out. A node down means killing one during the run. Srinivasan also notes that capacity added during a surge takes time to provision, join and rebalance, and that rebalancing consumes resources [6], so a fair evaluation has to include the rebalance window.
The obvious objection is that peak throughput is what a two-week evaluation can actually measure, and the rest is a modelling exercise carrying the vendor's assumptions. That holds for the data-growth condition. It holds less for the fault condition, since a p99.9 figure measured while a node is down is hard to produce unless the database really has the property it claims [10].
A pilot's data layer gets asked to carry later workloads. Srinivasan's example is a data layer chosen for a chatbot pilot that six months later is asked to support retrieval, personalization, real-time decisioning and an agent fleet at a significantly higher workload [13]. Agent loops make the tail harder to hide. One request may resolve in a single hop while the next runs 40 sequential retrievals [11], and in a sequential loop there is limited parallelism to absorb a slow step, so tail latency becomes end-to-end latency [12]. A slow lookup at step 31 produces a poor experience whatever the model answered [12].
A buyer who declines to measure variance still pays for it. Teams compensate with caches, read replicas and headroom, and each of those adds cost and operational complexity [14]. On the first of those, Srinivasan wrote: "A cache sitting in front of an unpredictable database is a bet on the hit rate." [20]
What to watch
- Published p99.9 and p99.99 figures from any vendor, Aerospike included, measured with a node deliberately down.
- Whether enterprise RFPs begin naming tail percentiles under an injected fault instead of peak sustained throughput.
- Whether agent frameworks add parallelism to retrieval loops, loosening the link between tail latency and end-to-end latency.