build1 publisher
A substring-match scorer made a weekend RAG build look 13 points worse than it was
Exact-substring scoring put a developer's 700-line RAG tool at a 65% retrieval hit-rate at k=3, 13 points below what a token-overlap scorer found. The bug also made extra retrieved chunks look worthless, and a 20-question test set added 15 points of noise.
Publishers:dev.to
Reality
- Evidence50
- Adoption
- Insufficient
- Hype gap+15
- Incentives
- Insufficient
- Confidence55