Build1 distinct publisher3 min readPublished
The ingest-sentinel project wrote dedupe, out-of-order folding and a five-minute late window down as explicit invariants. That is why one raw Prometheus query could separate a panel misconfiguration from real data loss.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Add the three raw counter values and you get 3,020 [20], which is the exact height at which the 20-count `late_rejected` series was drawn [16]. A series worth 20 was presented at about 151 times its own value [21]. The panel computed everything correctly; what failed was the reader's assumption that a line's height is its value. The default `{{label_name}}` legend then removed the cue that would have caught it, because `{{status}}` was never parsed and the labels came back truncated [17].
The diagnosis was cheap because the statuses partition the stream. `accepted` and `late_rejected` are mutually exclusive labels on one counter, `telemetry_events_ingested_total` [15], so both of them tracking total volume is not a state the service can occupy: either valid readings are being discarded or the metrics path is broken [13]. One query against the Prometheus API decided which [14]. That inference is only available if the exclusivity was specified in advance, with a hand-checkable expectation behind it: six shuffled events containing one duplicate, one out-of-order pair and one stale reading, yielding 3 accepted, 2 out-of-order corrections and 1 late rejection [6][7].
Worth noticing which invariant the load test actually stressed. The simulator generates sequential timestamps and then calls `random.shuffle()` across the whole batch [19]. Reordering is total, so the out-of-order counter legitimately ran past 8,000 [18], but lateness only occurs within the batch's own time span, so a shuffled batch of fresh readings barely reaches a five-minute threshold [24]. Exercising the late path means minting timestamps older than the window on purpose.
The five-minute window is a value that belongs to this deployment alone and will not carry over to yours as-is [5]. The write-up does not say what set it. In your system the floor is worst-case device clock skew plus the longest retry backoff you permit; set the window under that and legitimate retries land in the rejected bucket, aggregates come out low, and the totals still look like totals [5]. Late rejections were 0.67% of accepted events in this run [22], which is a share that sits comfortably inside the rounding on a daily figure.
The adoption cost is modest and concrete. One status label per outcome, so the counters stay mutually exclusive [15]. A rejected bucket stored outside the aggregates rather than dropped [5]. A dedupe key on event ID checked before the running sum is touched [3]. An update path where an older reading folds into history without displacing the latest value [4]. And a fixture small enough that you can do the arithmetic on paper, because the check you can perform by hand is the one that tells you what the dashboard is supposed to say [6].
Ranked by verification strength, evidence, and original report placement.
ingest-sentinel is a telemetry ingestion service built to explore how to preserve state correctness when sensor events arrive out of order, duplicated, or late.
A FastAPI ingestion endpoint accepts timestamped events from simulated IoT devices.
Invariant one, deduplication: the same event ID arriving twice cannot be counted twice in running aggregates.
Invariant two, out-of-order handling: an older event arriving after a newer one cannot overwrite the current latest reading, but must still be folded into historical aggregates and chronological logs.
Invariant three, late-arrival window policy: events older than 5 minutes are deterministically rejected, counted separately, and isolated from aggregates.
The author first built a 6-event ground-truth test: a handful of valid readings, one duplicate, one out-of-order pair and one deliberately stale event, delivered in shuffled order, small enough that the math was verifiable by hand.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · September 2, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
product
150 Jenkins masters, one control plane: the fix for CI sprawl was not a migration1 distinct publisher
build
A Prometheus that had written nothing for hours passed every health check1 distinct publisher
build
The network already knew: UPS and WAN state as keys a cluster can reconcile against1 distinct publisher
build
Pick a log anomaly detector on volume, latency and secrets, not on which one is smarter1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One self-reported bench run, but the key arithmetic checks itself
Everything traces to a single localhost run narrated by the person who ran it: the counter dump is pasted, not linked, and there is no repository, no dashboard export and no second pair of eyes. What keeps this from being an anecdote is that the load-carrying step is verifiable by a reader — 2,980 + 20 + 20 is precisely the 3,020 peak the panel drew, and the stacking behaviour is reproducible by anyone in two minutes. The surrounding figures fare worse: the 8,000-plus out-of-order arrivals are asserted, and the page contradicts itself once about which series that line belongs to.
No usage beyond the author's own bench
This is a side project pointed at its own simulator on a local Prometheus. Nothing in the reporting describes a release, a deployment, another user, or a production workload, so there is no adoption to measure — and inventing one from a 3,020-event self-test would be worse than saying so.
Modest overstatement, mostly in the moral
The narrative runs hot early — a stomach drop, a fork between a severe temporal drift bug and a lying graph — and lands on two default panel settings, which is a deflation the author performs honestly on himself. The overshoot is at the end, where one localhost run becomes a universal rule about never trusting the graph, and in what goes unexamined: 20 late rejections against a stated expectation of one or two per burst is a 10-to-20x deviation that the piece treats as vindication rather than a loose thread. A caption that credits the 8,000-plus line to accepted events, when accepted was 2,980, pushes the same direction.
Portfolio-shaped, no commercial stake
No vendor money, no product for sale, no launch to time: Grafana appears as the trap, not the sponsor. The pull that does exist is reputational — a build log whose hero moment is the author catching his own lazy rationalization rewards a tidier arc than debugging usually has, which is plausibly why the awkward 20-versus-one-or-two arithmetic never gets revisited and why the fix is billed at two minutes.
Mechanism I would bet on, numbers I would re-run
Split verdict. The diagnosis — stacked series plus an unparsed legend making a tiny counter look like total volume — is the kind of thing a reader can reproduce on their own dashboard before lunch, so it survives having exactly one source. The measurements around it do not travel as well: one machine, one simulator, self-reported counters, an untested five-minute window and a page that disagrees with itself about which line climbs past 8,000.