Intercom wants support teams to ship AI agents the way engineers ship code

Intercom has announced Evals and Releases, which together with its existing Monitors feature are meant to let teams test changes to the Fin agent before they go live, roll them out with control, and score every live conversation afterwards ^[1]. The interesting part is not the features but the standard they imply: Intercom is asserting that a customer-facing AI agent is a piece of probabilistic software and should be released like one ^[11].

Look at the mapping. Evals is a regression suite: a named group of multi-turn simulated conversations built from your own inbox, grouped by theme such as refund requests, escalation rules or tone of voice, each scored pass or fail against criteria you set, and re-run after any change to catch unexpected regressions ^[2][6][17]. Releases is a staging branch with a canary and an A/B arm: bundle changes to content, Procedures or Guidance, test them with Evals, then publish to everyone, ramp traffic gradually, or run an A/B test against Fin's current configuration ^[3][10]. Monitors is production monitoring, checking every live conversation against your standards and flagging the ones that fall short so they become the next round of fixes ^[4]. Intercom calls the loop "Eval-driven delivery" and says it is the discipline its own AI research team uses ^[5]. The correspondence with ordinary software release practice is close enough to be the whole point ^[18].

The argument for it is a drift argument. Intercom says Fin reasons through each conversation as it happens, so it can answer the same question in different ways while customers ask it in countless ways, producing thousands of scenarios no team can validate by hand ^[11]. Editing one help article or adding one Procedure can change behaviour somewhere else without anyone noticing ^[13]. And correctness is not one field: Intercom lists whether the right Procedure triggered, whether the right content and data sources were used, whether handoff happened at the right moment, and whether it sounded like your brand ^[12]. The number that carries the case is Intercom's own: at the scale some of its customers run Fin, a 1 percent regression could affect thousands of conversations a day ^[14]. Taken literally, that puts those accounts at roughly a hundred thousand conversations a day or more ^[19].

Two things to keep your eye on. First, the grader. A Simulation is scored by deterministic checks plus an LLM judge, with the judge assessing things like whether Fin replied and said what it needed to say, whether a Procedure fired, and whether a data connector was called ^[7][9]. A pass rate is only as good as agreement between that judge and your best reviewers, and the announcement as supplied gives no figures on that, nor on pricing or availability ^[20]. Second, the labour claim. Intercom says every support team should be able to test and monitor rigorously without a developer or ML specialist ^[16], and says existing oversight tooling was not built for this scale ^[15]. Someone still has to write the criteria, curate simulations from real conversations ^[8], and decide what a fail means.

The durable effect is definitional. Once a vendor ships a staging environment and a traffic ramp for a support bot, "we updated the macro and watched the queue" stops being an acceptable answer, whether or not you run Fin.