Build1 distinct publisher3 min readPublished
Uber reported 169:1 on 200TB of Spark logs a day by separating static templates from variable fields before anything was written to HDFS. Whether that ratio transfers depends on how few call sites your lines come from.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The arithmetic tells you what the 169:1 is measured against. The projection Uber's team rejected was linear: ten times the retention, ten times the bill, $180K to $1.8M a year [4][3]. HDFS billed per byte, and nobody expected cross-job deduplication to help. Divide the rejected figure by the published ratio and you get about $10,650, which is the "roughly $10K a year" the writeup reports [1]. The ratio carries the whole cost outcome, and it is measured against the already-compressed corpus rather than raw text. Per day of retention, that is $60K before and $333 after [2].
A generic compressor cannot get there by tuning. zstd finds repetition with a sliding window over bytes and entropy-codes the remainder, without knowing which bytes are a timestamp and which are an order ID [8]. CLP splits the line before compression happens: the static template that comes from the call site, the variable values that differ per occurrence, and a dictionary per field type [9]. What reaches the compressor afterwards is a small set of unique templates plus one column per field, and a column of order IDs is far more repetitive on its own than it was interleaved with the prose around it [10]. The dev.to post calls this the Parquet intuition applied to log semantics [11], which is fair: group like with like, then pick an encoding per group.
Which is where 169:1 stops being portable. That number describes Uber's Spark logs, roughly 250,000 jobs a day emitting up to 200TB from a fixed set of call sites in a fixed set of binaries [2]. Template extraction pays in proportion to how many lines share a template. For the ratio to move to your cluster you would need the same shape: a handful of call sites producing most of the volume, variables that are numeric or low-cardinality, and not much free text. A stream dominated by stack traces with distinct frames, user-supplied strings, or JSON carrying UUIDs and base64 has less scaffolding to factor out. The figure worth measuring is unique templates per million lines in your hottest stream, which you can count locally in an afternoon.
The adoption side goes unpriced. The writeup reports the storage outcome and confirms it was lossless, with no sampling and no dropped DEBUG lines [6][7]. It does not quantify CPU at ingest, decode latency on read, or what happens to every tool that assumed it could grep a text file. Those costs land on the platform team, and they are the ones that decide whether the encoder is worth wiring into the write path.
The framing the post argues against, "we need a cheaper log platform," is worth taking seriously as a diagnosis rather than a slogan [12]. Uber's three-day retention was a budget expressed in days [3]. On this evidence, when someone proposes cutting retention, the encoder is the cheaper thing to examine first, and the number that settles the argument is one you own.
Ranked by verification strength, evidence, and original report placement.
In 2022 Uber's engineering team published the numbers behind its Spark log storage problem.
Uber's Spark platform was generating up to 200TB of logs a day across roughly 250,000 jobs.
Log retention on HDFS was capped at three days, not because three days was enough but because it was what the storage budget could absorb; engineers kept asking for a month.
Extending retention naively at existing compression would have taken the HDFS storage bill for these logs from $180K a year to $1.8M a year.
Uber's team adopted CLP, the Compressed Log Processor, originally developed as academic research at the University of Toronto, and restructured how logs were encoded before they hit disk.
The reported result was a 169:1 compression ratio, HDFS costs down to roughly $10K a year, and retention extended to a full month.
Distinct publishers with included, body-backed reporting in this cluster.
1 article · September 5, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Object or file storage: the wrong pick stays quiet until inode exhaustion1 distinct publisher
build
Cloudflare tests compressing its cache at rest rather than buying more disk1 distinct publisher
build
The codec runs last: what a hand-written ORC parser shows about columnar speed1 distinct publisher
leadership
Uber sizes its internal support load at 45,000 questions a month1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single relay, self-consistent numbers
Every figure reaches us through one dev.to write-up of an Uber engineering post from 2022. Uber's own account is not in our material and no second publisher has checked it. What lifts this above a bare relay is that the numbers constrain each other: the rejected $1.8M projection divided by 169 comes out near the $10K reported, so the ratio and the cost drop describe one measurement rather than two unrelated boasts. Where the post reaches for outside proof, in CLP's published benchmarks, the available text gives comparisons without a single number and then breaks off.
One production user, four years old
There is a real deployment at real scale here, and exactly one of it. Uber's Spark logs at 200TB a day is not a demo, but the disclosure dates to 2022 and no other user appears in this reporting. Nothing is said about CLP's license, its release history, or what the encoding step costs in CPU where it now sits in the write path, so the breadth behind the one data point is unknown.
Real number, stretched scope
The 169:1 is Uber's own production figure and the cost arithmetic holds, so the headline is not inflated on its own terms. The overreach is in scope. The ratio is a function of how few distinct log statements produce a fleet's lines, and the piece presents it as what schema-aware encoding does rather than what it did at one call-site distribution. The searchability argument goes further, calling compression a strict upgrade on the strength of CLP's design intent to support search without full decompression, with no latency measurement anywhere in the reporting.
Nothing disclosed either way
Our material shows no relationship between the author and Uber, the CLP project, or any log storage vendor, and no promotion of a product to buy. That absence is not a clean bill of health, only an absence: a personal developer-platform post carries no disclosure section, so there is nothing here to weigh.
Traceable but unverified
The specific claims are attributable and mutually consistent, which is more than most cost-saving stories manage. Against that sits a single publisher, a four-year lag, an original engineering post absent from our material, and a text that ends mid-sentence. Confidence in the mechanism is higher than confidence in the ratio transferring.