Build1 distinct publisher3 min readPublished
The pipeline logged a healthy count every hour for a full day and delivered nothing downstream, because items were recorded as handled at the moment they were seen rather than the moment they left the queue.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The write order is not sloppiness. A dedup ledger is an idempotency record, and the safest moment to write to it is before the work starts. Write on completion, and a crash mid-batch replays items you already sent. Write on arrival, and the ledger never lies about what has been seen. Plenty of implementations pick arrival, because that is the cheap durable point, and then the ledger is a seen-set with a field called `done` [5].
Then the cap runs. It exists to keep the pipeline from overwhelming downstream services, and items over the line are meant to wait for the next cycle [6]. In this order they do not wait. They are already in the ledger, so the next cycle's dedup step recognises them and skips them [5]. The trim is a delete wearing the label of a defer. The second commit message says it without decoration: "pipeline: stop burning jobs at the cap, and stop starving the best source" [9].
Volume is why nothing tripped. Spread 175 items across a full day of hourly runs and the average run reports a little over seven [1]. Seven is the count an operator reads as normal and moves on.
The rest of the dashboard was measuring liveness. The self-test log returned a pass on four checks in 3318ms [11], and the mail watcher returned `"ok": true` [12]. The restart counters were louder and less useful: cto-aipa at 99 restarts in two days is roughly one every 29 minutes [3], and algom-stream at 55193 in fourteen days is about 164 per hour, one every 22 seconds [2]. A counter moving that fast has stopped being an alarm and become weather.
The check the author says was missing is items ingested against items delivered, with an expected delta [14]. The word doing the work there is "expected". Once you have a cap, the delta is never zero, so the useful form is a conservation statement: ingested equals delivered plus deferred plus deduped, with every term readable for the cycle. That forces the ledger to hold a state rather than a boolean, claimed versus delivered, which is the same constraint the landed fix satisfies by running the cap before anything gets marked [8].
Whether this reproduces in your pipeline comes down to two lines you can find in an afternoon. Locate the ledger write and locate the trim, then check which runs first, and whether the trim re-enqueues the overflow or drops it. If the overflow is pushed onto the next cycle's input before the ledger is touched, you have a defer. If not, you have this incident with different variable names.
One more constraint is worth reading, because it explains why the ordering bug survived a day. The operator's own notes file states that Cursor Cloud, Cursor Desktop and Claude Code all work the same repo, none of them can see each other's chats, and the only artefacts all of them read are HubSpot and that file [16]. The author says this missing shared context makes incidents like this harder to debug [17]. An invariant that spans two steps has to live somewhere both editors will read it, which is what a wiki entry titled "the queue marked 175 items done before anyone read them" is actually for [15].
Ranked by verification strength, evidence, and original report placement.
The root cause was a dedup ledger stamping items as handled before a processing cap was applied, which meant 175 items were marked as done without ever being processed.
The consuming agent, VibeJobHunterAIPA_AIMCF, was starved because downstream systems received zero items; the problem was a misattribution of done status rather than a lack of data.
The operator's PM2 process list shows 8 processes online, including cto-aipa with 99 restarts in 2 days and algom-stream with 55193 restarts in 14 days.
A newly added data source ran on schedule for a full day, logged a healthy count every hour, and delivered nothing at all downstream; the system reported success while the output queue was empty.
The author characterises the event as not a silent failure but a loud success that produced nothing.
The dedup ledger exists to prevent redundant processing: if an item was already seen it would be marked done and skipped.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 30, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
AI-written code fails the same four ways, and every gate you own reports green1 distinct publisher
build
Superpowers makes spec-driven work a precondition, then ships it to twelve harnesses1 distinct publisher
build
NVIDIA put a number on agent skills: 300+ verified, two harnesses, baselines under 50/1001 distinct publisher
product
A 2x LLM bill is not a bug report: token spend is an observability problem1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Specific, but one custodian
The specifics are unusually concrete for a personal postmortem — two commit hashes with messages and dates, quoted log lines, a wiki entry title, verbatim NOW.md text. They are also all readings from one machine that only the author can take, and the author says outright that the ingest-versus-delivery figure is 'not measured yet', which means the central 175-versus-zero comparison was reconstructed from two log sets rather than a standing counter. Nothing is contradicted; nothing is corroborated either.
One repo, one operator
The blast radius and the install base are the same thing: a single self-run stack. Two commits shipped the same day, six processes' worth of telemetry is disclosed, and no other user, team, or dependent system appears anywhere in the reporting. The pattern may generalise widely; the deployment does not.
Quieter than its own numbers
The headline is punchy but the body under-claims rather than over-claims: no grand thesis, an explicit refusal to blame the restart counts, and a plain admission that the detecting metric is still unbuilt. If anything the piece buries its loudest figure — 55,193 restarts in 14 days, one every 22 seconds, filed as someone else's problem. Pushing the other way, the reordering is presented as the solution with no post-fix delivery count to show it held, so the resolution is asserted a little ahead of its proof.
Credibility-building self-publication
This is a cross-post from the author's own site, and the agent it defends — VibeJobHunterAIPA_AIMCF — is the author's product, named throughout. A well-told postmortem is reputational capital for a solo builder, which is a real reason to publish and a real reason the framing favours 'careful engineer catches subtle bug'. Against that: no sponsor, no product pitch, no benchmark, and the author airs numbers that make the stack look fragile.
Plausible, unverifiable
The mechanism is coherent enough that an experienced pipeline engineer would recognise it on sight, and the artefacts are named rather than gestured at. But everything rests on one self-published account with no second reading, no linked repository, and no after-state, so our reading of what actually happened is only as good as the author's own log inspection.