Build1 publisher3 min readPublished
Three judges score every answer in a RAG sweep, including one from the generator's own vendor
A dev.to write-up fixes retrieval at 200-word chunks and cosine top-5, then swaps three embedders and five generators across 100 questions and four topics to find out whether its own evaluation conclusions hold.
The Engineer · Build desk
What happened
- A dev.to write-up inverts an earlier framework comparison: the RAG pipeline is now the frozen, hand-rolled part, and the models are the variables, with three embedders, five generators, three judges and four topics.
- The frozen pipeline uses 200-word chunks with 20-word overlap, cosine similarity, top-5 passages to the generator and one shared prompt, with no frameworks, no reranking and the production database untouched.
- Each judge returns two things per answer: a factual correctness score from 0 to 100 against the reference, and a true/false verdict on whether every claim is supported by the retrieved passages.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Checking that the conclusions are model-independent costs 600 generated answers and 1,800 judge gradings per sweep, and every model or judge added to a slot multiplies both.
- capability Because DeepSeek's answers are the only ones graded entirely by judges from other vendors, the panel has an internal control for whether a judge inflates its own family.
- decision Anyone reusing the eventual generator ranking has to decide whether their own chunk size and top-k are close enough to a no-reranking top-5 cosine setup for the numbers to carry.
The isolation rule is what makes this usable as evidence. Three embedders and five generators define fifteen possible pairs, and six of them run [2]. Pipelines p2, p3 and p6 keep the baseline embedder and change only the generator; p4 keeps Claude Sonnet and changes only the embedder; a test enforces that structure [13]. p5 changes both at once, so it is flagged and never used for a single-variable claim [14]. The generator slot therefore gets three clean comparisons against the baseline, and the embedder slot gets one [3]. Gemini embedding appears only in p5 [4]. The write-up asks which slot matters more, the embedder or the generator [20], and the grid answers that with three swaps on one side and one on the other.
Every answer is graded three times, by a Claude judge, a GPT judge and a Gemini judge, and the panel includes the judge from the generator's own vendor on purpose [15]. All three get an identical plain-JSON rubric with no vendor-specific structured output [16]. Each returns factual correctness from 0 to 100 against the reference, plus a true/false verdict on whether every claim in the answer is supported by the passages retrieval returned [17]. Four of the five generators have a same-vendor judge on that panel; DeepSeek does not, so its answers are the only ones scored entirely from outside the family [9].
Twenty of the hundred questions have answers that are verifiably absent from the corpus, and the only correct response is that the documents do not cover this [5]. Several of those sit right next to material the corpus does mention [6]. A model that answers one has answered from memory [21]. The WHO nutrition fact sheets are in the set because pretrained opinions are strongest on that topic, so refusing to invent an answer costs the most there [9].
At 10 questions, one lucky or unlucky answer moved the score by 10 points [4]. At 100, it moves the score by 1 [1]. That resolution costs 600 generated answers per full sweep and 1,800 judge gradings, which produce 3,600 scores [5]. The post does not report what the run cost. Ten of the hundred questions are the SEPA rulebook questions carried over verbatim from the earlier golden set, so 90 are new [8][7].
The first bug in the record is in the grading code. "the groundedness judge caught me before it caught any model," the author wrote [18]. In its first version the judge received bare chunk texts while the generator had seen the same passages labeled; the published sentence breaks off there [19]. When the judge reads the context in a different form than the generator did, the verdict measures the presentation difference along with the answer.
The write-up states its pass condition directly: "If the behavior and the numbers survive swapping every model slot, then the conclusions belong to the system rather than to some vendor's checkpoint, and I can replace any model later without the evaluation story collapsing" [3]. It also says "I am not looking for a winner here" [22]. Whatever the scores turn out to be, they will describe a pipeline with cosine top-5 retrieval and no reranking [7]. Five chunks of 200 words puts roughly 1,000 words of retrieved context in front of the generator [8]. A stack with a reranker, or a larger top-k, hands the generator different material, and the generator ranking would have to be measured again there.
What to watch
- Whether the published per-judge scores show an offset when a judge grades a generator from its own vendor.
- Whether Gemini embedding ever gets a pipeline that holds the generator fixed, which would make the embedder question a three-way read.
- Whether the 20 refusal traps are split evenly across the four topics, given that nutrition is where refusal is hardest.