Atlassian swapped its metrics engine behind the same StatsD address on 100,000 hosts

The contract Atlassian froze is narrow. Service owners see StatsD over UDP arriving at an address and their metrics showing up in the backend, and everything between that packet and long-term storage belonged to the platform team ^[7]. Purpose-built Collector distributions went in at each of four stages, collection, ingest, aggregation and forward, so work on one would not disturb the others ^[8]. On the collection swap the app side stayed identical. "Day one, no team noticed," the team wrote ^[11].

A 30-day month at a 99.95% target allows about 21.6 minutes of unavailability ^[21]. Atlassian's engineers described the obvious alternative, tearing out the old pipeline and having every team re-instrument on the OTel SDK, as "a pipe dream: a multi-year org-wide slog on a pipeline that can't take an outage with a real chance of dropping the exact data alerts fire on" ^[6].

The two sidecar numbers have different denominators. If saving about 3.9% of per-service CPU is roughly a 30% cut in sidecar cost, the sidecars together were taking about 13% of CPU on those services before one of them was removed ^[22].

Ingest had a constraint the other stages did not. Aggregation is stateful, so every datapoint for a time series has to hit the same aggregator, and ordinary load balancing cannot guarantee that ^[13]. Metric load per service follows a long tail, so hashing on (service, environment) put whichever shard owned the biggest service permanently under load ^[14]. Hashing on streamID, the identity of a single time series, spreads one large service across the pool while keeping each series on one shard ^[15]. Per-shard CPU went from a couple of tall bars beside idle replicas to an even distribution, with real off-peak scale-down and no more hot-shard pages ^[16].

220 million out of 4.8 billion is 4.6%, a 95.4% reduction, which the post rounds to 96% ^[19]. Spread over roughly 100,000 hosts, the input works out to about 48,000 datapoints per host per minute ^[20]. Most of those metrics use delta temporality, and nothing upstream aggregated deltas the way Atlassian's users expect, so the team wrote its own delta aggregation processor ^[18]. One of the reasons given for leaving gostatsd was that "every clever thing the OTel Collector community shipped was one more thing we'd eventually rebuild by hand just to stay level" ^[5].

Two conditions decide whether this route is open to a team planning the same move. The first is whether the new engine can receive the old protocol at the point of collection: Atlassian turned on StatsD and OTLP receivers together, so nobody had to swap clients before the work started ^[9]. The second is whether the interface teams depend on stops at a protocol and an address. Where it reaches into client-library behaviour, the re-instrumentation is the migration and no address can hide it. Atlassian also had a prerequisite most shops lack: its tracing team had run the OTel Collector as its pipeline core and host-metrics sidecar for years, so production-readiness at that scale was settled before the metrics team started ^[10].