Loading today’s stories
Build1 distinct publisher2 min readPublished
Two sprints of tuning put the savings in label cardinality and per-tenant limits, not the setting the team started with.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A global retention setting disappoints for a reason that lives on the write path. In Loki, every unique combination of label values creates a stream, and every stream gets its own chunk [4], so a label like `request_id` or a pod IP does not merely add data, it multiplies the index [4]. Retention shortens how long each of those objects survives. It has no opinion on how many you made. That is why the ordering in the dev.to write-up is the load-bearing advice: run `logcli series --analyze-labels` over the last 24 hours before editing any retention value, and if one label returns thousands of distinct values, the retention config is the wrong file to be sitting in [5].
Getting `trace_id` and `request_id` out of the label index and into structured metadata, supported from Loki 2.9 onward [10], is not a toggle. The bill arrives as pipeline work: anything routing those fields through label relabeling in Promtail or Grafana Alloy has to be rewritten to target structured metadata instead [11].
Then there is the part of the invoice that is not bytes at all. At high ingestion volume, object storage cost tracks request count, and small chunks mean more PUT and GET calls; the write-up puts the saving from raising `chunk_target_size` into the 1.5 to 4MB band at roughly a third of PUT volume [8]. That saving is paid for in ingester memory, since a longer `max_chunk_age` means fewer flushes, and the team reported a couple of ingester OOMs during a traffic spike before it raised memory limits [9].
The example override block rewards arithmetic. team-payments gets 2160h for compliance, team-web gets 168h [13], a 12.9x spread inside one cluster that no single global number can express [1]. The dashboard figure is sharper: a 30-second auto-refresh against 90 days of cold chunks works out to 2,880 scans a day from one panel [2], which is why the 72h `max_query_length` sitting under team-web's retention line [13] does more for that cost than the retention line above it. Blocking the query is the cheaper half of the fix, because the spend that panel generated was request and egress driven [12], [8].
The same arithmetic leaves team-web paying to store 168 hours of logs while querying 72 of them by default, so 57 percent of the retained window is unreachable without an override [3]. That is a defensible trade for high-volume, low-value logs, but it is a trade, and it only exists because retention and queryability were separated per tenant. The metric worth putting on a wall is unique streams per tenant, not days retained.
Ranked by verification strength, evidence, and original report placement.
The author spent two sprints on Loki retention tuning last quarter and reports the actual fix had almost nothing to do with the retention_period setting they started with.
Loki supports per-tenant retention overrides in runtime_config, and they reload without restarting ingesters.
Payments audit logs and frontend debug logs sat under the same 30-day policy for over a year; splitting them into separate tenant overrides cut storage for the noisy tenant by 70% while extending retention for the compliance-sensitive one.
Every unique combination of label values creates a new stream, and every stream gets its own chunk, so a label like request_id or a pod IP multiplies index size rather than just adding data.
The recommended cardinality audit is logcli series --analyze-labels --since=24h '{}'; if it returns thousands of unique values for a single label, retention tuning will not fix the problem.
Setting retention_period without enabling the compactor is a silent no-op: Loki accepts the config and deletes nothing. After shortening retention for a tenant, storage stayed flat for two weeks.
Follow any of these and your For You feed starts watching them — no settings page required.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Concrete configs, one unverified reporter
The article supplies unusually specific artifacts — exact compactor, ingester and overrides YAML, a logcli audit command, named Loki versions for structured metadata — which makes the mechanisms checkable. But every quantitative outcome is a single practitioner's recollection with no baseline, no measurement methodology, no dashboards, and no second source in the cluster to corroborate or contest. Mechanism claims (streams per label combination, compactor-gated deletion) are internally consistent and independently plausible; the result claims are not verifiable from what is supplied.
One undisclosed-scale deployment
Adoption evidence is limited to a single self-disclosed production cluster where the described changes were actually applied, plus the author's assertion that the underlying features (per-tenant runtime_config overrides, compactor retention, structured metadata in Loki 2.9+/3.x) are available in shipped Loki releases. There is no second deployment, no vendor usage data, no community survey, and no indication of cluster size or ingest volume.
Headline outruns a single anecdote
Modestly overstated. The framing ('cut nothing' vs 'cut 70%') generalizes one tenant's unbaselined storage reduction into a rule, and the sharpest claim — that the fix had almost nothing to do with retention_period — is undercut by the article's own advice, which is largely about retention machinery (compactor enablement, per-tenant retention_period values, delete delay). Offsetting the inflation, the author volunteers real tradeoffs: ingester OOMs, non-free pipeline migration, delete-delay lag, and that retention_period is not guaranteed erasure. The gap is presentation-level, not fabrication.
Low commercial pull, reputational upside
The piece is a cross-post from the author's own site to dev.to, with a listicle headline and no product being sold, no sponsorship marker, and no affiliate or vendor call to action; it points readers to official Grafana Loki documentation rather than a commercial alternative. The identifiable incentives are audience-building and practitioner credibility, which favor dramatic before/after numbers — consistent with the unbaselined 70% headline. No employer or vendor affiliation is disclosed, so a hidden interest cannot be ruled out from the supplied material.
Plausible mechanisms, unverified outcomes
Confidence is capped by single-source dependence: one publisher, one author, one undisclosed-scale cluster, and no contradicting or confirming coverage. The configuration-level and mechanism-level claims are stated precisely enough to act on and test cheaply, which raises confidence in the guidance; the quantitative results (70% storage, one-third fewer PUTs) should be treated as directional only.
build
An empty Detected fields panel is a Loki capacity incident that has not billed yet1 distinct publisher
build
Ten agents, 500 files, one naming rule: the migration that did not need parallelism1 distinct publisher
build
Eight containers against a variable APM bill: what the self-hosted stack actually costs you1 distinct publisher
build
S3 to Lambda is async and at-least-once: the 3% that vanished after eight quiet months1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 26, 2026