Raindrop replays real production traffic against every pull request to test agent changes

"If we're having an issue like a build failure or agents stuck in a loop, we see that issue in Slack," said Bani Singh, an AI engineer at Vercel ^[13]. On a Monday, the product is a message in a channel that somebody has to read and then decide about.

Simulations replays real production traffic and a team's existing test cases against a proposed change, then runs Raindrop's anomaly detection over the results ^[4]. Replay only works on flows users have already exercised. A tool path added in the same pull request you are testing has no traffic behind it, so what you get is regression coverage on behaviour that already exists.

Raindrop said conventional evals rely on test cases written in advance, so they mostly catch the failures a team already thought of ^[5]. Take that as a vendor's account of the practice its product replaces. The operational claim underneath it is the placement: Simulations runs on every pull request ^[4].

"Agents now run for hours, call thousands of tools, and handle real money, real health data, and real customers," said Zubin Koticha, Raindrop's chief executive ^[1]. The company lists unnamed Fortune 100 enterprises in healthcare and logistics among its customers ^[10]. It points to OpenAI research on deployment simulation, which regenerates responses to de-identified production conversations with a candidate model ^[8]. Raindrop describes its own product as replaying real production traffic, and the announcement did not disclose whether that traffic is de-identified first ^[16].

The pace argument comes from METR: the length of tasks agents can finish on their own roughly doubles every seven months, and a single run can now last days and involve thousands of tool calls ^[6]. Twelve months of that is 2 raised to 12/7, or 3.28, so the longest task an agent finishes unaided gets about 3.3 times longer over a year ^[7]. A hand-written suite does not grow at that rate.

Money keeps landing nearby. groundcover raised $100m in July for AI-era observability, Scaled Cognition took $100m from Khosla in June, and Harvey bought Guardrails AI this month ^[15]. "Raindrop treats agent failure as a detection problem, the way a security company would," said Reid Christian, a general partner at CRV ^[11]. Bucky Moore of Lightspeed said bad agent behaviour will become catastrophic, and pointed to high-stakes settings such as defence ^[12]. The hiring matches the framing: engineers who built fraud detection models at Robinhood and anomaly detection at Square, plus security engineers from Segment, Semgrep and Socket.dev ^[14].

For a given team it comes down to the days of production traces you actually retain for the flows that touch money or health records, because replay against a thin corpus tests the paths your busiest users take and little else. Then there are the minutes one engineer spends deciding whether a flagged behaviour change is wrong or merely different, multiplied by the pull requests the team opens in a week. Simulations is in research preview ^[17].