DoorDash threw away a working safety classifier. The measurement that came first is the reusable part

Nine percent is a generous ceiling on "small single digit", and it is enough to do the architecture arithmetic. Of 4 million daily messages, at most 360,000 would ever need an expensive opinion, and at least 3.64 million would not ^[15]. At the 2 to 10 second latencies Pereira quotes for her team's LLM calls ^[7], that escalated slice works out to roughly 8 to 42 model calls in flight at any moment, averaged across the day ^[16]. Peaks will be worse than the average, but that is a fleet you can reason about, and it is why the cheap layer is not a compromise: it turns an unaffordable per-message problem into a small one you can afford to do well.

The latency arithmetic is less forgiving. A sub-second decision budget against 2 to 10 seconds of measured model latency is over budget by 2x at best and 10x at worst ^[17], and no gain in classification quality returns any of that time. Model selection is not the lever here. Routing is.

Which is what makes the rewrite the more useful half of the talk. The published transcript stops as the small classifier's three jobs are introduced and never lists them ^[19], so the replacement's internals are not in this record. What Pereira says she wants the audience to keep is a pattern rather than an implementation, one she claims applies to almost any AI use case ^[3]. On the evidence she has already laid out, that pattern is an ordering: get the base rate off your own traffic first, then size a cheap filter to it and let the expensive model see only what the filter cannot settle. None of that is bound to a particular model, which is exactly why a system that worked could be thrown out ^[2] without taking the design with it.

The bill for the pattern lands in one place. "Right almost all of the time" ^[11] on a safety product means the remainder is an abusive message delivered, to two parties whose relationship runs somewhere between six and forty minutes and who have no history to fall back on ^[13]. The talk gives no false-negative rate for the cheap layer. Against a stated goal that every message reaching its destination has been classified as safe ^[5], that is the figure worth asking for, and it is a decision about tolerable residual risk rather than a modelling detail. Verbal abuse is described as a meaningful share of incidents on the platform, and feeling safe is treated as a product metric on equal footing with being safe ^[14], which means the misses are measured in both.