Build1 distinct publisher3 min readPublished
A multi-agent orchestrator held its cross-site correlation window in process memory. It fired live on the first try, for the wrong reason, and the fix was a shared record in Firestore.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A missed correlation makes no noise, and that is the whole of the problem. The orchestrator's job is to spot the same risk signature at more than one site inside a time window [6], and its evidence for "more than one" was a list held in process memory [2]. Route two readings to two instances and the second one reads an empty list, then concludes, correctly given what it can see, that nothing correlated [4]. There is no exception to chase and no retry to inspect. The output of a broken correlation window and the output of a genuinely quiet fleet are the same output, which is why the live run came back clean on the first attempt [1].
So the feature was only correct while Cloud Run declined to scale. The write-up is explicit that light traffic often keeps requests on one warm instance, and that neither the single local process nor the reused instance is a guarantee [3]. That inverts the usual reading of a green demo: the more traffic the fleet attracts, the more often the correlation path quietly returns nothing. It is also the same rule the project already applies to the model layer, where Gemini drafts a summary of a decision the deterministic code has already made and never re-derives the risk numbers [18]. Nothing incidental should own a decision, and instance affinity is incidental.
The fix went to the store that was already there. Recent-event state moved into Firestore, which held each site's isolated survey history, rather than into a second mechanism introduced for this one bug [14]. Verification was two live runs with fresh station IDs, a new correlation notice appearing both times against the shared record [15]. Worth being precise about what that establishes: state now survives the process boundary. Two separate sequential runs are not a contention test, and the source describes nothing that forced two instances to write the same record at once [15]. Since a site-agent publishes on escalation instead of the orchestrator polling Firestore on a timer [17], two escalations in flight together is the arrangement the design actively invites.
The same defect class turned up in the data. The original spec named Bori, Ogbogoro and Onitsha [10]. Ogbogoro's source paper has been unreachable for months on dead DNS [11], and Onitsha only ever had a rough estimated range from a search summary, used to fake a synthetic sample [12]. That leaves one of three sites with real observed history [2], while the headline feature needs at least two [6]. The roster is now Bori, Choba and Etche, all surveyed and independently published [13]. The author's description of the alternative is the useful sentence: a demo that would have looked identical and meant nothing [19].
The scoring is what makes this more than a matter of taste. Architectural discipline is 30% of the Fortified Enterprise Fleet track [8] and proving the thing runs on Google Cloud is a separate 30% [9], so 60% of the marks [1] rest on claims a single passing run does not settle either way. The deployment half would have scored the same before the rewrite. The other half is the one that gets read by someone examining the state-management story, which is where a bug that only looked fixed comes apart [21].
Ranked by verification strength, evidence, and original report placement.
It worked in local testing because there is only one process, and worked live because Cloud Run under light traffic often reuses the same instance instead of spinning up a second one; the author states neither is a guarantee.
If traffic patterns shifted and two readings landed on two different instances, the second instance would not know the first existed, and a correlation that should fire would silently not fire.
The author swapped the roster to Bori, Choba and Etche, described as fully real, surveyed and independently published.
Recent-event state moved out of process memory into Firestore, the same store already holding each site's isolated survey history, so that no second separate way of sharing state was introduced for this one problem; every instance now reads and writes the same shared record.
The author verified the fix with fresh station IDs submitted live, twice, on two separate runs, watching a genuinely new correlation notice appear both times against the shared store rather than against whatever instance happened to still be warm.
The correlation notice fired live against the deployed service on the first try: three sites, same anomaly type, inside the time window, caught and logged by the orchestrator.
Follow any of these and your For You feed starts watching them — no settings page required.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed but single-source and self-reported
The mechanism, failure mode, and fix are described with unusual specificity for a build log, and the author distinguishes what is load-bearing from what is not. But everything rests on one first-person post with no linked repository, logs, test output, rubric, or site-survey citations, and the key falsifying test (forcing two concurrent instances) is not reported.
Single-author hackathon prototype in live deployment
There is a real deployment on managed Google Cloud services with disclosed live verification runs and a passing test suite, but usage is confined to the author's own hackathon submission: no external users, teams, customers, or downstream deployments are reported.
Slightly understated relative to its own findings
The framing is self-critical rather than promotional: the author downgrades a successful demo, states explicitly that Gemini is not structurally load-bearing, notes that neither local nor live single-instance behaviour is a guarantee, and closes by admitting other early successes remain unchecked. The mildly dramatic headline is the only inflationary element, and it is supported by the body, so claims sit at or slightly below what the reported evidence would license.
Disclosed hackathon-entry incentive on a self-assessed build
The article is explicitly written to enter Google's All Things Agentic Hackathon, in a track the author says scores architectural discipline and proof of Google Cloud deployment at 30% each, and it evaluates the author's own system using vendor services. That is a strong incentive to foreground architectural rigour and Google Cloud usage. The incentive is disclosed openly, which mitigates but does not remove it.
Moderate: internally coherent, externally unverified
Confidence is limited by single-source, self-reported evidence and a disclosed hackathon incentive, and lifted by the account's internal consistency, its specificity about mechanisms, and its explicit scoping of unproven and non-load-bearing elements. The transferable engineering lesson is more reliable than any individual project detail.
build
A self-healing scraper that must prove its repair against twelve records that cannot move1 distinct publisher
security
Google's reference agent approved a $10,000 refund on a $149 order, on purpose1 distinct publisher
security
Mandiant found 100 high-severity bugs in two days. Plan for the other side doing the same.1 distinct publisher
product
Pixel 11 Pro XL: the upgrade you can feel is shutter lag and a dictation tool1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 26, 2026