Build1 publisher2 min readPublished
Mallika Rao locates the hard problems of production recommenders at the boundaries between components
Mallika Rao argues in an InfoQ presentation that production recommenders get hard at the boundaries of a six-stage request loop. Her case rests on years of practice. It shows teams where to look but does not measure how engineering effort should be split.
The Engineer · Build desk
What happened
- Mallika Rao argued in a presentation published by InfoQ that the hardest part of building recommenders is the end-to-end system, not the model alone.
- She named the operating constraints that system has to meet: latency, cost, observability, experimentation, customer trust and compliance.
- She said one of the biggest mistakes she has seen is organizations treating recommenders as siloed problems and isolated models.
- She said today's AI debates over feedback loops, evaluation and online learning are ones recommender teams have been having since before generative AI went mainstream.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision A team that accepts the boundary claim has to give evaluation, experimentation and feedback plumbing their own owners and roadmap, and plan them as product work alongside the model.
- cost Much of the cost of an adaptive recommender comes after launch and falls on whoever owns latency, observability and compliance for the live system.
- precedent Teams building LLM agents can draw on recommender practice in feedback-loop design and evaluation. Rao's point is that it predates the problems those teams now face.
Rao's model of one recommendation request has six stages after the user arrives [3][4]. The system retrieves as many candidates as it can, ranks them, performs inference, evaluates outcomes, produces the list, and feeds the outcome back into itself [3]. Laid out that way, each stage looks like something one team can own and benchmark. She places the problems worth that attention somewhere else. "The interesting problems are actually happening at the boundaries of these components, and not inside each of these components necessarily," she said [5].
The last stage shows why. The list a user sees becomes the outcome fed back into the system [3]. A change to ranking therefore alters what the rest of the loop learns from. In my view this is what her boundary claim means in practice: a ranker can pass its own offline eval and still degrade the loop. Rao said these systems are "a big feedback system" and "distributed systems," with "a lot of experimentation going on under the hood" [7].
She gave the older approach credit. Earlier recommenders ran on handcrafted heuristics, business rules and carefully tuned ranking functions, and she said that "surprisingly, most of those systems worked really well" [8]. Anyone about to replace one might keep that sentence next to the rollback plan. Those systems were limited by sparse signals, shifting user intent, scale and slow adaptation [9]. Embeddings, contextual signals, multistage retrieval and ranking pipelines addressed those limits. In her account they also brought a new class of operational challenges [10].
Her evidence for the budgeting conclusion is her own experience, built over years in search, discovery, personalization and candidate generation [12]. The opening of the talk includes no incident counts and no split of engineering time between model work and pipeline work. For the conclusion to carry over, a team's system has to resemble the ones she describes: adaptive, with a feedback stage that closes quickly. A recommender retrained offline on a slow schedule changes fewer boundaries at once. I'd expect model work to take a larger share of effort there. Rao tied the new operational problems to "quickly evolving user behavior as you're interacting with your product" [10].
What to watch
- Any measured breakdown, from Rao or others, of whether production recommender incidents start inside a stage or at the boundaries between stages.
- Whether LLM agent teams adopt evaluation and experimentation tooling from recommender practice, the parallel Rao drew.