A substring-match scorer made a weekend RAG build look 13 points worse than it was

The miss that exposed the scorer was a question about Liechtenstein ^[5]. The answer key said "Switzerland and Austria." The top-ranked chunk described a country "bordered by Switzerland to its west and by Austria to its east" ^[5]. Retrieval had found the right passage. The substring check failed it because the exact phrase never appears in one piece ^[5].

A second bug had the same shape. The corpus wrote "around 60,000" and the answer key wrote "60000". Stripping punctuation split the first into two tokens that could never match the second ^[7]. Joining digit groups before normalizing recovered one more point ^[7]. That squares with the corrected curve: 78% after the overlap fix, plus one, is the 79% reported at k=3 ^[13].

The k curve is where the bug nearly changed the product. Retrieval pulls the top k chunks by L2 distance ^[3], so the top 10 for any question always contain its top 3. On a fixed question set, hit-rate can only rise with k ^[14]. Under the substring metric it rose two points between k=3 and k=10. Under the fixed metric it rose four ^[15]. "The broken metric didn't just report a wrong number, it pointed me at the wrong decision," the author wrote ^[16].

Both slopes are small. The 65% starting point matches the 100-question run ^[9]. If the curves used that set, the broken curve's rise was two questions and the fixed curve's was four ^[17].

Sample size is the second problem. The post puts the 95% interval at 20 questions at roughly plus or minus 22 points ^[10]. The normal approximation agrees: 1.96 times the square root of 0.25/20 is about 0.22 ^[18]. At 100 questions and a 65% rate, the same formula gives about 9 points ^[19]. Scoring all 918 questions in the dataset would bring it under 3 points ^[20]. A hit-rate check needs only a question embedding and a vector lookup, with no call to Claude ^[25]. "If your eval set is small enough to hand-write in an afternoon, it is small enough to tell you whatever you want to hear," the author wrote ^[23].

The corrected scorer brings its own error. A rule that credits any single chunk holding 80% of the answer's content words ^[6] will also credit a chunk that names both countries for some unrelated reason. The post does not report checking the new scorer against hand-labelled retrievals. In my view the phrase and digit fixes are correct. I would still treat 79% as an estimate with unknown bias until a few dozen hits are labelled by hand.

The best engineering in the post is one line of output. Each query prints its sources and marks the cited ones with an asterisk ^[12]. In the Uruguay example, the answer cited two of the three retrieved chunks and ignored the third ^[12]. The author wrote that this gap "is the first place you see retrieval quality without running an eval at all" ^[21].