Raindrop replays production agent traces against synthetic services on every pull request

Replaying a cached API response stops working as soon as the agent acts on the world. That is Raindrop's own account of the problem: a write to a database, a refund, a search across a repository that has changed since the trace was captured, or a call to a tool that did not exist when the trace was recorded ^[10]. Raindrop says its answer is stateful synthetic copies of the services the agent touches, plus generated data shaped like the customer's environment ^[11]. The copy has to accept the write and then hold it for the remaining steps of the trajectory.

The verdict is a statistical judgement about trajectories. Raindrop says Simulations reruns production traffic and existing tests against the proposed change, then applies its anomaly-detection system to the resulting trajectories. It reports regressions, cost changes, output drift and tool errors in the pull request ^[9]. An intended improvement and a regression both arrive as a change in behavior. Separating them is the detector's work, and a gate that flags every deliberate improvement gets switched off within a week.

The case for replaying traces at all is coverage. Handwritten tests cover the failures engineers already expected. Agents also break through an unanticipated sequence of tool calls, stale context, a changed prompt, or a model substitution that alters behavior several steps into a task ^[8]. The founders described the same gap when they entered Y Combinator's Winter 2024 batch selling monitoring infrastructure. Conventional application monitoring catches a crashed service or a failed network call. It misses an agent that completed a request incorrectly, forgot an instruction, entered a loop, or took a technically valid action against the user's intent ^[17][18].

OpenAI has published the nearest thing to a public result for this method. In June it described deployment simulation over de-identified production conversations, used to forecast undesirable behavior before a model release ^[12]. It reported that realistic simulations gave it a better view of how often behaviors occurred than selected test prompts, while acknowledging that rare failures still require targeted evaluations and red-team testing ^[13]. That result carries over to an application agent only if the failure appears in your production traffic often enough to land in the replayed set. It also has to reproduce against a synthetic database or payment system instead of the real one ^[6]. OpenAI was simulating conversations with its own models; the traces Raindrop replays include writes to services it has to stand up itself ^[11].

Adopting this in the order the product implies means instrumenting production first, because the test corpus is your own traces, and collecting them is what Raindrop's original monitoring product did ^[5]. The launch account does not include a price. The founders reached the same conclusion the expensive way, building Sidekick, a coding agent and VS Code extension that had paying users and no effective way to tell why it failed after deployment ^[15].