Build1 publisher2 min readPublished
Stamping a trap chunk id into every negative test turned two false passes red
An embedding swap to bge-large made two long-failing negative tests pass because the retriever stopped surfacing the trap chunks at all. A chunk-id check in CI gives the suite a third verdict, test did not run.
The Engineer · Build desk
What happened
- A team swapped its embedding model from text-embedding-3-small to bge-large last February on the assumption that a better embedder would mean better retrieval-augmented generation.
- The golden set stayed green after the swap, and two negative tests that had been failing for months started passing.
- The retriever had stopped pulling the trap chunks, so the model never saw the trap and its polite refusal matched the expected answer for a question the documents do not support.
- With a CI check asserting the trap chunk id is present in the retrieved set, the same two tests went red, and the team rolled back the swap and fixed chunking before re-validating.
- Because chunk ids move whenever documents are re-chunked or re-parsed, every pipeline change now triggers a re-stamping pass over the golden set before any verdict counts.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Without a retrieval assertion, a passing negative test cannot separate a model that refused from a retriever that missed, so a green suite certifies less than the pass count suggests until a third verdict exists.
- decision An embedder swap becomes a re-validation queue: tests still stamped with the old model have to be re-run before their verdicts are usable.
- cost Whoever owns ingestion pays roughly 20 minutes per pipeline change to re-stamp trap chunk ids, and the author treats that as non-negotiable.
- capability Asserting a named chunk id lets teams test refusals without tuning a similarity threshold, which matters because the overlapping distributions leave no cut point to tune.
The retrieval check runs first in the code, and that ordering is the design. In the published `ci_eval.py`, retrieval runs, then `if trap_chunk_id not in retrieved_chunk_ids` sets the verdict to "test did not run", and only the `elif` compares the model's answer to the expected one [9]. A refusal that matches the expected string `"not in documents"` cannot reach the pass branch unless the trap chunk was in the retrieved set [9][8].
The assertion flipped the two tests. The model's output did not change when the assertion was added; the verdict did, from pass to red [17]. According to the author, the new embedder was worse at semantic similarity for those specific traps, and the fix was a rollback plus a chunking change before any re-validation [16][7].
Two conditions have to hold before this transfers. The trap has to live in one chunk you can name at authoring time, which is how the id gets stamped [4]. And chunk ids have to be recomputable, because they move on every re-chunk or re-parse [10]. If your negative case is an absence spread across a document set, there is no single id to assert on, and the test stays a claim about the answer.
The assertion also checks membership: `not in` is a set test, with no position compared [20]. A trap chunk returned last in a large candidate list counts as retrieved even if a reranker or a context budget drops it before the model reads it. The failure the author describes sits upstream of that, at the retriever [3].
The second stamp handles the slower failure. Each test records the embedding model it was last validated under, and after a swap every line still carrying the old tag is listed in the CI summary under "unverified under current embedder" until someone re-runs and re-dates it [6]. The author credits that list with stopping two shipments of broken retrieval [12].
The threshold approach is the alternative this replaces. The post cites Amritpal Singh, whose refusal threshold had no effect because the similarity distributions for answerable and unanswerable questions overlapped [13]. Cosine similarity cannot separate "about the topic" from "answers the question", since both score high [14]. Asserting a named chunk id sidesteps the search for a cut point that the distributions do not offer [13].
The post credits Serhiy Kucherenko with the finding that "the corpus decides more than the model does", and frames its own failure as the retrieval boundary breaking [15]. All of this comes from one write-up. It does not disclose how many tests the golden set contains [18]. Two tests changed verdict on one swap [2][7].
What to watch
- Whether the author publishes retrieval numbers for the bge-large traps beyond the two tests that changed verdict.
- Whether a case appears where the trap chunk is retrieved but dropped by reranking or a context budget, which the membership check would still score as a run.
- Whether the 20-minute re-stamping estimate holds as the golden set grows.