Build1 publisher3 min readPublished
A replayed eval job carrying a production write key committed its late result to the shared store
Sharing one idempotency key let a late shadow eval result commit to production, according to a dev.to design review written for MonkeyCode. Its fix rewrites the key at admission and separates the queues too, at the cost of diffs that are harder to read.
The Engineer · Build desk
What happened
- A free execution server replayed an evaluation job that still carried a production write key, according to a dev.to author describing an incident last month.
- The model stream returned late, the shadow worker wrote its payload to the shared store, and production then skipped a body its planner still expected.
- The key was shared on purpose: the common path stamps the same idempotency key onto the free server job because that makes diffs easier.
- The proposed fix has the control plane rewrite a shadow job's key at admission, before any model call, and send execution to a separate shadow store.
- The post was prepared as part of MonkeyCode's product outreach and uses MonkeyCode's free model access and free server option as its shadow plane.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Teams that separate the planes can no longer diff shadow and live outputs on a shared key, so each comparison needs its own mapping from rewritten keys back to live ones.
- exposure A tool endpoint called by both paths stays a route from the shadow plane into production until its handler checks which plane sent the request.
- decision The key rewrite has to be enforced at the control plane's admission step, because a forgotten rewrite there is the only listed failure that reaches live state.
The shared idempotency key meant the store could not tell the two planes apart. In the author's account, production skipped its own write because deduplication worked as designed. The write it deduplicated against came from the shadow plane [2]. The post's failure sequence shows how one stall produces that write. The control plane admits job-7 with the key live:tenant:debit, the model plane stalls after buffering its proposal, the client retries, and the free server accepts a second worker [8]. Worker A's late completion commits amount=10. Worker B then commits amount=12 under the same live key [8].
A second worker on one key needs only two assumptions, and the post states both: the planner retries after a stall, and delivery to the execution plane is at least once [9]. "Two planes that share a key namespace are one failure domain, even when their dashboards look completely separate," the author wrote [15]. On the name: "I would not call it a shadow, because a second writer with a friendly name is still a writer" [14].
The fence in the post's diagram is one line. The live commit log stays untouched unless prefix == live AND admitted_plane == live [7]. I like the second clause. Take a job admitted as shadow that kept its live: prefix because the rewrite was skipped. It still fails the admitted_plane check [7].
The best section of the post sorts faults by who can change state [11]. The model plane can stall, truncate or return a late completion, and that should only invalidate a proposal [11]. The free server can accept twice, die mid-write or reject for capacity, and that should only touch the shadow store [11]. Only one listed failure reaches live state. It is the control plane admitting a duplicate after forgetting the rewritten key, and the author calls that leak the dangerous one [11].
Separate stores do not close the path if both planes call the same tool endpoint. The author wrote that a shared tool URL without a plane header is another shared write key [10]. "I treat that address as a door, not as a domain, until the handler checks the plane" [16].
The author accepts the cost. The planes must not share queues or compensation topics either, "even when that makes diffs harder to read" [5]. Once the key is rewritten, shadow and live results have no common key to join on, so any comparison needs its own mapping between them. I would pay that whenever the live key names a debit, as live:tenant:debit does in the post's own example [8].
The evidence is one account. "I am reviewing a design rather than reporting a load test," the author wrote, and declined to invent quota, hardware or longevity figures for the free option [13]. The post does not say how many live writes the incident affected. It was written for MonkeyCode's outreach, but the author wrote that the invariant holds for any shared execution tier [12].
What to watch
- Whether MonkeyCode's free server option documents a plane header or a separate key namespace, since the fence depends on handlers checking the plane.
- Measured quota or capacity figures for the free tier, which the author says can go into the fixture without changing the fence [c17].
- A worked example of diffing shadow and live outputs after key rewriting, the cost the post concedes.