Scoring 300 candidates in 35 milliseconds forces the LLM out of the request path

Divide the 35 millisecond ranking line by the roughly 300 candidates it scores and the per-item cost is about 0.117 milliseconds ^[2]. The 300 candidates are scored in one batched pass over the shortlist. For the number to transfer to another stack, item embeddings have to exist before the request arrives, the user vector and recent events have to return from cache inside the 5 milliseconds allotted, and the ANN index has to be resident in memory ^[9][10].

The five stages sum to 75 milliseconds against the stated 100 millisecond p99, leaving 25 milliseconds for network and queueing ^[1]. The post's opening uses a tighter deadline, 80 milliseconds, and on that figure the slack is 5 ^[1][3]. Neither figure has room for a generative call in the request path, so the LLM work moves elsewhere: embeddings generated at ingestion, query understanding cached per distinct query, evidence selection decided asynchronously and reused ^[10]. When a model genuinely has to run per request, the author keeps it to a shortlist of tens, behind a timeout, with a deterministic fallback ^[11]. A strip that degrades to a popularity-ranked list is fine; one that returns nothing because a model call hung is a broken page, the post says ^[12].

Those timings are the post's own worked example ^[9]. The two 2026 papers it cites support a narrower point. RecoChain, given as arxiv 2604.25787, proposes unified generative retrieval and ranking and is then evaluated against classic top-k ranking metrics, because the measurement requirement does not go away ^[3]. RRCM, arxiv 2605.07129, frames LLM recommendation as retrieval-and-reasoning, where the model decides when to pull more evidence and optimization still runs against the final ranking reward ^[4].

The most common data-layer mistake, according to the post, is recording what users clicked without recording what they were shown ^[20]. The impression row it proposes is seven fields wide: request_id, user_id, item_ids, positions, ranker_version, retrieval_source and a timestamp, with interactions joined back on request_id ^[13][4]. Without that row you cannot compute a click-through rate, correct for position bias, or evaluate a new ranker counterfactually against logged traffic ^[14]. Online A/B testing becomes the only way to learn anything, and the post calls that slow and expensive ^[14]. The per-item retrieval_source is the diagnostic field: when quality drops, it tells you whether the collaborative path or the embedding path produced the bad candidate ^[15].

Recall@k asks whether the item the user eventually chose made the shortlist at all, NDCG@k asks how the shortlist was ordered, and a ranking model cannot fix a candidate set that never contained the right item ^[16].

Matching two sides of a marketplace runs the same four layers with two changes. Mutual constraints mean a candidate has to clear both sides' filters, so the retrieval layer runs twice and intersects ^[17]. Supply also exhausts: recommending the same top-rated contractor to 400 buyers produces 399 disappointments, so the ranking layer needs a fairness or throttling term ^[18]. Pure item recommendation gets by without one.