Loading today’s stories
Build1 distinct publisher3 min readPublished
A generator writes with equal confidence whether its context is right or wrong, so the only readout that separates the layers is a labelled eval set you own, scored on the retrieval side before an answer exists.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
The arithmetic behind that demo is the part worth carrying around. At a hit rate near 30% [3], about seven of every ten queries hand the model context that does not contain the correct chunk [1]. The generator does not degrade gracefully there. It writes with the same confidence it uses when the chunk is right [5], so distinguishing grounded from plausible means opening the source and checking it, per answer, which nobody does [6]. In that room, the presenter's blink was the only instrument.
Layer separation follows from a mechanical fact, not a preference. If the retriever returned the wrong chunk, no amount of prompt or model tuning repairs the answer [8], which is why a single "did the answer look good" score cannot tell you where to spend the sprint [9]. Hence two families of metric: retrieval scores computed before the LLM sees anything, generation scores computed against the context it did see [20]. The routing rule is cheap to apply once you have both sides [19].
The rank metrics are where the retrieval side earns its keep. Hit rate is binary at k [10]; MRR grades where in the list the right chunk landed, with position 1 scoring 1.0 [12]; precision@k grades how much noise came with it [13]. Take the author's own example, a correct chunk at position 5 with four irrelevant ones above it. That is precision@5 of 0.20 [2] and a reciprocal rank of 0.25 [3], against 1.0 on both for a clean hit at position 1. Identical hit rate, and one of the two systems is paying four chunks of tokens and feeding the model material that confuses it [13]. NDCG grades partial relevance, and the author's advice is to skip it because hit rate and MRR cover 95% of decisions [14].
The 80% floor at k=5 [11] is a number from someone else's corpus, and it transfers only under conditions worth stating. Hit rate as defined counts whether *the* correct document appeared [10], singular, so the floor describes single-hop questions with one gold chunk per query. If your answers need three chunks stitched together, an 80% single-chunk hit rate is clearing a weaker bar than the same number would clear for the author. The floor also assumes k=5 is the context budget you actually ship. None of that rescues the demo: 30% sits 50 points under the floor [4] before you even ask what k the logs were measuring.
The bill is labelling. Every number above needs a query set with the correct chunk known in advance [10], rerun as a fixed set rather than sampled fresh each time, which is the only way the failure distribution becomes visible at all [7]. On the generation side, faithfulness measures the fraction of claims in an answer supported by the retrieved context, and the author reports it as the metric that would have caught a client's wrong-dosage incident [15]. Answer relevancy is separate, because an answer can be entirely faithful to the context and still not address the question asked [16]. Context precision asks how many retrieved chunks were needed [17]; context recall asks whether the answer was derivable from what came back, which is the hit-rate question viewed from the generator [18]. Two of those four are really retrieval measurements wearing generation clothes, which is fine, as long as you do not count them as coverage of both layers.
Ranked by verification strength, evidence, and original report placement.
Fluent output hides broken retrieval: the LLM is a confident text generator, so a wrong chunk in the context produces an answer that sounds exactly as good as a right one, and the model writes with equal confidence either way.
A human reader cannot distinguish an answer grounded in the right source from a plausible-sounding one without checking the source themselves, and nobody does that for every answer.
A demo question that happens to retrieve well says nothing about real questions that do not; the only way to see the failure distribution is to run a fixed, labelled set repeatedly.
If retrieval returns the wrong chunk, no amount of prompt or model tuning fixes the answer.
A single "did the answer look good" number tells you nothing about where to fix things; retrieval and generation must be scored separately to know where to invest.
Hit rate / Recall@k is defined as the fraction of queries for which the correct document appeared in the top-k retrieved results.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 28, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
product
Green dashboards, invented refund policy: the case for a separate AI eval layer1 distinct publisher
build
Don't start at the model layer: classify inputs by reliability, then let RAG wait1 distinct publisher
build
Netflix's LLM ranker won 0.006 percent. The number that matters is 40x fewer labels.1 distinct publisher
build
A million 'AI slop' reports in two weeks, and not one of them measures accuracy1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Definitions solid, numbers unwitnessed
Cut this reporting in two and the halves score very differently. The taxonomy — Recall@k, MRR, Precision@k, faithfulness, context precision, context recall — is standard and stated precisely enough to implement from the page. The empirical spine is one person's memory: the 30% hit rate comes from logs no reader can see, the wrong-dosage incident is an unnamed client, and the 80% floor arrives without a dataset. With dev.to as the only publisher, there is no second reading of either half.
No deployment trail
Nothing in this reporting amounts to adoption. There is no release, no version, no benchmark run, and no team on the record saying it scores these metrics in production. The one system described is an anonymous meetup demo, and the 100–200 pair eval set is a recommendation rather than a practice anyone here has shipped.
Thresholds firmer than the data
The overall posture is deflationary — distrust your demo, measure less confidently — which pushes against inflation. What tips it the other way is the certainty attached to unmeasured constants: 'the single most important number in all of RAG', below 80% 'full stop', NDCG relegated to the last 5% of decisions. Reasonable engineering heuristics dressed in benchmark clothing they never earned.
Consultant's ledger, lightly showing
The author writes as a practitioner with clients and earlier posts to point back to, and the two most vivid details — the demo he was called in to review, the dosage error a client shipped — are exactly the sort that build a reputation for auditing other people's pipelines. No product, vendor or price is being sold on the page, so this reads as credentialing rather than salesmanship.
Checkable where it teaches, opaque where it proves
We can test the taxonomy against ordinary retrieval practice and the worked numbers against themselves, which covers most of what a reader will actually act on. We cannot test anything that gives this account its urgency, and with one publisher there is no cross-check to reach for. The text also stops mid-sentence in the tooling section, so the advice on scoring at scale never arrives.