BuildNot yet confirmed elsewhere1 publisher2 min readPublished
EVE Online traces a 20-year-old single-shard game with head-based sampling only
EVE Online runs 100% head-based sampling with no tail sampling, Nicholas Herring said in a talk published by InfoQ. The team blends deterministic and exponential-moving-average sampling to balance trace fidelity against cost.
The Engineer · Build desk
Key figures from Nicholas Herring's talk: a 12 billion item count that includes every bullet in EVE Online's universe; about 500,000 market transactions per day; and 100% head-based sampling, with no tail-based sampling.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Item count, including every bullet in the universe | 12,000,000,000 items | 9 | |
| Market transactions, about | 500,000 per day | 13 | |
| Head-based sampling | 100% | 2 |
What happened
- Herring called exponential moving average sampling, the dynamic half of the blend, a "blessing and curse."
- EVE runs as a single shard with no regional servers, so every player shares one universe on one server.
- Shared play forces the game to synchronize, replicate and keep data transactional, a problem Herring compared with FinTech.
Why it matters
- constraint A rare failure or slow request survives in EVE's traces only if the sampler happened to keep it at the start, because nothing downstream can promote a dropped trace.
- decision Teams weighing head against tail sampling first have to decide what they must be able to reconstruct. EVE's answer fits traffic that includes player expression the system need not care about.
- precedent Head-only sampling with a dynamic component now has a named production example, but the evidence is one team's account in one talk.
Head sampling makes its keep-or-drop call when a trace starts, before anything about the outcome exists. A head sampler cannot pick out the slow or failed traces afterwards. Tail sampling decides once the trace is complete. [14]
Herring does not dispute the cost. He said the choice "makes a lot of people clutch their pearls and gasp" and that the team is "buying in a lot of those tradeoffs." [3] The transcript excerpt stops during the scale discussion, before any rate, window or threshold appears and before it says which traffic the deterministic rules cover.
The scale figures set the volume. Twelve billion items across about half a million market transactions is 24,000 items per transaction on average [13]. The average is loose. Herring said the item count includes "literally every single bullet that's in the universe," and that the talk has to deal with tracing "all of the bullets and lasers and missiles flying around in the ecosystem" [9].
He also gave a reason some loss is tolerable. Video games, he said, do "a lot of stuff that the system doesn't really need to care about," and he used waving at another player as the example [11]. In our view that context is what makes 100% head-based sampling defensible here. A dropped trace of a wave costs nobody anything.
The choice transfers only to a system where a dropped trace is cheap to lose. A payment path that has to explain every failure would not qualify. We'd expect the deterministic half of EVE's blend to cover the paths that cannot be dropped.
What to watch
- The rest of the transcript: whether it states sampling rates, the moving-average window, and how the deterministic rules combine with the dynamic ones.
- Whether EVE's team reports incidents where a trace dropped at the head would have shortened the diagnosis.
- Herring's separate talk on running EVE as a single shard, which he said covers the technical challenges of that design.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence40
- Adoption50
- Hype gap+5
- Incentives
- Insufficient
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Nicholas Herring presented how EVE Online adopted observability into a system that is 20 years old and has newer mechanisms evolving underneath it.
- [2]
EVE Online's system is 100% head-based sampling and does not use tail-based sampling.
- [3]
Herring said the 100% head-based approach "makes a lot of people clutch their pearls and gasp, because we don't use tail base sampling which means we're buying in a lot of those tradeoffs."
- [4]
EVE Online blends dynamic and deterministic sampling techniques.
- [5]
Herring described exponential moving average sampling for dynamic sampling as a "blessing and curse" and as "the crux of the tricks that we're using."
- [6]
Herring said the aim is to find a balance between "good enough for fidelity and cost."
- [7]
EVE Online is a single-sharded MMO: there is no North American, APAC or Europe server, and everybody experiences the same universe at the same time on one server.
- [8]
EVE Online's market has about half a million transactions per day, representing about 12 billion items moving around the ecosystem.
- [9]
Herring said the 12 billion item count "also counts literally every single bullet that's in the universe," and that the talk has to cover tracing "all of the bullets and lasers and missiles flying around in the ecosystem."
- [10]
Herring said multiplayer games create shared experiences, so they share FinTech's problems in different ways: they must synchronize a lot of data, keep it transactional, and replicate it.
- [11]
Herring said video games do "a lot of stuff that the system doesn't really need to care about," giving waving at somebody as an example of a player's expression in the virtual world.
- [12]
Herring said he has a separate talk on the technical challenges of running EVE Online as a single shard.
- [13]
Across the market, 12 billion items over about 500,000 transactions per day is roughly 24,000 items per transaction on average.
- [14]
A head-based sampler decides whether to keep a trace when it starts, before its outcome is known, so it cannot select slow or failed traces after the fact; tail-based sampling decides after the trace is complete. EVE Online's choice of head-only sampling therefore forgoes outcome-based selection.
Sources
1 independent publisher whose own reporting we read for this story.
- infoq.comPresentation: How Eve Online Leverages Head Based Sampling to Observe "Fun"
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Game backend engineeringFollow
- Distributed tracing samplingFollow
- ObservabilityFollow