Uber's security experts withheld the Slack channels until Genie's retrieval improved

The failure mode Uber describes sits in retrieval, not generation. Since RAG was introduced by Lewis et al. in 2020, the recurring problem in question-and-answer setups has been that ambiguous or context-poor queries pull the wrong documents, and once irrelevant material is in the window the model produces errors or invents an answer ^[11]. Uber's response was to wrap agents around the pipeline, running steps before and after retrieval to sharpen both what gets fetched and what gets written ^[10], while rebuilding the offline document-processing half of the system separately from the near-real-time half ^[13]. A platform that makes deployment free does not remove the cost of accuracy; it moves that cost into the pipeline and the people who grade it.

The evaluation set is thinner than its headline count suggests. More than 100 queries spread across more than 40 policy PDFs works out to roughly two and a half questions per document ^[15], which is a sample of what on-call actually gets asked rather than coverage of the corpus ^[5]. That is a reasonable way to build a gate, and it is worth naming what kind of gate it is: the veto sat with the experts whose queue the bot was meant to shorten ^[7].

Uber reports the outcome as a relative 27% increase in acceptable answers and a relative 60% reduction in incorrect advice ^[8]. A skeptic will say relative movement from an undisclosed baseline is a publication number, and the skeptic is half right, because the account gives neither the starting acceptance rate nor the absolute figure after the change ^[14]. What the arithmetic does bound is the ceiling: 27% relative on a baseline of 70% acceptable lands near 89%, and on a baseline of 50% it lands near 64% ^[17]. The other half of the claim is easier to read plainly, since a 60% relative cut leaves 40% of the previous volume of incorrect advice still going out in the channel ^[16].

The goal Uber sets is the one to hold it to. Near-human precision is defined here as experts relying on the bot for most queries without worrying about misinformation in the engineering security and privacy domain ^[9]. That is a claim about behaviour, and the published evidence is graded answers rather than any observed decline in double-checking. Whether the experts actually stopped auditing replies is not in the record, and neither is the latency or token cost that extra agent calls put in front of an on-call question ^[14].

For anyone costing an internal copilot this quarter, the split is between what the platform amortises and what it does not. Infrastructure spreads across every domain that adopts it, and Uber's framework was built precisely to make that spread cheap ^[3]. The golden set, and the expert hours to write it and score the results, is bought again for each domain that wants channel access ^[5]. Uber's own numbers point at the second purchase as the source of the improvement, which makes evaluation capacity the constraint on how many of these bots an organisation can responsibly run at once.