Build1 publisher3 min readPublished
Grading a retriever starts with hand-labelling 500 queries against 100,000 chunks
Shrijith Venkatramana's embedding evaluation guide treats retrieval as a ranking problem scored with Recall@k, MRR and NDCG. Every figure in it is a worked example, and the labelling is the real bill.
The Engineer · Build desk
What happened
- A developer guide on dev.to opens with a vector database returning 20 documents in 15 milliseconds and still producing a terrible RAG answer, and locates the hard part in whether the right documents rank near the query.
- It names three metrics to score that: Recall@k for whether relevant material was retrieved at all, MRR for how quickly the first relevant result arrived, and NDCG for whether the most useful results sit at the top.
- Hindi queries against English documentation, Latin-script product names inside Hindi sentences, and internal jargon absent from training data are raised as questions the guide says a benchmark must contain.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The entry fee for any recall number is human labelling: 500 queries, each with its answering chunks named by hand out of 100,000, plus a second set that has to stay untouched during tuning.
- constraint The retrieval window caps answer quality. When the chunk holding the answer sits outside the top 10, a better generator has no path to it, so the model upgrade buys nothing on those queries.
- decision Scoring two candidate embedding models on identical labelled queries makes a swap decision arguable with a figure. The alternative is picking the model that looks more semantic.
- contradiction Cross-lingual and jargon breakage enters the guide as a hypothesis to test. The recall figure is missing, so a team can copy the test design but cannot use the piece to price the loss.
Cosine similarity is a ranking function. The guide writes it as cos(q, d) = (q . d) / (||q|| * ||d||), and notes that when the vectors are normalized to unit length it reduces to the dot product [4]. Venkatramana writes that the application does not care whether document A scores 0.83 and document B scores 0.79; it cares whether A appears before B [5]. The score exists to order candidates.
He anchors that in 2002 work by Kalervo Järvelin and Jaana Kekäläinen on graded relevance, where a document can be irrelevant, somewhat useful or highly useful [6]. Their motivation was that retrieval systems produce too many results, so evaluation has to reward systems that put highly relevant material near the top [6].
The worked example is five relevant documents, a top-5 of [X, A, Y, Z, B], and Recall@5 of 2/5 = 0.40 [10]. Three of the five relevant documents never enter the window [14]. The first relevant document sits at rank 2 [15]. On the case where the needed chunk misses the top 10 entirely, Venkatramana wrote: "A stronger language model does not magically fix missing context." [12]
Those numbers arrive as suppositions. The corpus is introduced with "Suppose you have a documentation system with 100,000 chunks", the query volume with "Imagine 1,000 evaluation queries" [17]. The guide does not report a measured system, so the 15 ms against 20 documents [1] comes with no index type, corpus size or hardware. For that latency to transfer to your deployment you would need the same index structure, the same shard count and the same filter predicates at query time, all of which the guide leaves out.
The procedure is where the cost sits. Five hundred real user queries over a 100,000-chunk corpus, with the documents that actually answer each one identified by hand [7]. For "how to rotate api keys" that was three chunks: doc_1842, doc_7119, doc_9320 [7]. Three relevant chunks in 100,000 means a random top-5 draw would be expected to surface 0.00015 of a relevant document [16]. Ordering is the whole job, and the only way to know whether your model does it is a human deciding relevance 500 times.
The payoff the guide claims is comparison: the same queries scored across candidate models, instead of an assertion that Model A "looks more semantic" [8]. In my view that is the strongest part of the argument, because it makes an embedding swap falsifiable before it ships.
The cross-lingual material is weaker as evidence. Hindi queries against English documentation, a product name in Latin script inside a Hindi sentence, and internal terminology that never appeared in the model's training data are posed as questions [11], and multilingual queries and typos appear in the list of ugly cases a benchmark should include, alongside short queries, vague queries, queries with multiple correct documents and queries whose answer is buried in a large document [9]. No recall figure is attached to any of them. The instruction that closes that list is the one I would argue for on its own merits: "keep a separate test set that you do not use while tuning the model" [9].
What to watch
- A measured Recall@k delta for Hindi-script queries against English documentation would turn the guide's cross-lingual questions into a number a team can budget against.
- Whether embedding vendors publish Recall@k and NDCG on a held-out set next to their latency figures.
- Whether anyone attaches a cost to fine-tuning an embedding model instead of labelling more queries.