Build1 publisher2 min readPublished
Per-byte metering on 50KB records pushed Perplexity from DynamoDB to a home-built Rust store
Perplexity moved its search serving tier from DynamoDB to CobbleDB, an in-house Rust store, and cut median batch reads from 31.4ms to 5.60ms. The cost case carries to other retrieval teams only at similar payload sizes and request rates.
The Engineer · Build desk
What happened
- Retrieval for language models pulls full chunked passages and dense embeddings, so Perplexity's average record is about 50KB.
- Above 200,000 requests per second, DynamoDB's usage-based pricing became unsustainable because AWS meters every byte transferred, according to InfoQ.
- Reprocessing jobs for new chunking schemes or embedding models wrote directly into DynamoDB and contended with live user reads.
- Perplexity split storage into three systems: Pillar for durable state, Lorry for batch aggregation and CobbleDB for low-latency serving.
- In production, p99 batch-read latency fell from 123ms to 24.2ms and p90 fell from 56.7ms to 9.77ms.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Under per-byte metering, storing full passages and embeddings under each key sets the size of the read bill, so record shape becomes a pricing choice as well as a retrieval one.
- capability Staging reprocessing output as S3 batch files lets Perplexity re-chunk or re-embed its corpus without writing to the nodes that answer live queries.
- constraint The latency design depends on asynchronous replicas, so it fits only reads that tolerate lag; a workload needing read-after-write consistency cannot adopt it as published.
One answer pulls about 5 to 6MB out of the store [1]. A query touches 100 to 120 page keys [3] at roughly 50KB a record [4], read as 5 to 12 parallel batches [2]. Production traffic runs above 200,000 requests per second, according to InfoQ [5]. If each request carried one average record, that is a floor of about 10GB a second through a meter that charges per byte [3]. If a request is a 10-to-20-key batch, multiply by ten to twenty [4].
The only cost figure in the account is storage, down at least 20 percent [2]. InfoQ does not give a number for read spend, the reason cited for leaving [5].
The latency gain holds across the distribution: 5.6 times at the median, 5.8 at p90 and 5.1 at p99 [5]. The changes the article describes are placement and routing choices, and none of them is specific to Rust. They target the causes of tail spikes the article pins on DynamoDB: uncached reads, cross-zone hops and lagging replicas [6]. CobbleDB's router sends reads to replicas in the caller's availability zone and hedges a slow read to a replica on another node [12]. Each node answers a batch with one RocksDB MultiGet call [13], served from memory-mapped cache on local NVMe [11].
The write path is the part I would copy first. Pillar keeps versioned tables on YTsaurus over mechanical drives and commits crawl updates and export queues together in one transaction [9]. Lorry packs exports into partition-aligned files in S3. CobbleDB workers pull those files on their own schedule, so a re-embedding run [7] never writes to a serving node directly [10].
CobbleDB also drops distributed transactions and synchronous consensus. Each of the three replicas per partition [11] applies updates at its own rate [14]. The stated justification is that search serving tolerates slight replication lag [14]. A workload that needs each read to reflect the last write cannot take the design as published.
The synthetic result, steady throughput to 500,000 requests per second at payloads up to 100KB [16], describes Perplexity's own hardware and key distribution. A benchmark written by a store's authors against their own traffic is the friendliest kind there is. For it to transfer, a team's reads have to be batched lookups by hashed key [12]. The team also takes on node lifecycle work that a managed database handled, the first trade-off InfoQ lists [17]. A per-byte bill shrinks in proportion to payload and rate. I'd expect the operations cost of three replicas per partition to shrink far more slowly, and below Perplexity's scale that difference favors staying managed.
What to watch
- A published read-spend comparison between DynamoDB and CobbleDB, the direct test of the per-byte pricing argument.
- Replica lag during a full re-embedding run, when every partition ingests new S3 batch files at once.
- Production latency as traffic moves from 200,000 toward the 500,000 requests per second seen in synthetic tests.