Build1 publisher3 min readPublished
Normalizing text before the hash lookup finds several times more contaminated eval rows
Train/test contamination gets into a fine-tuning set through merges, augmentation and re-scrapes, and the first two checks for it are complete hash lookups with no threshold to tune. The approximation only enters at the near-duplicate stage.
The Engineer · Build desk
What happened
- A dev.to walkthrough argues eval examples reach the training set through ordinary steps: merged public datasets, augmentation before the split, re-scrapes, and training files that keep growing.
- It splits detection into three checks with three costs: exact byte comparison, normalized comparison, and near-duplicate matching, of which the first two are complete hash lookups with nothing to tune.
- Near-duplicate detection shingles each text into word 5-grams, estimates Jaccard similarity with MinHash over a common 128 permutations, and generates candidate pairs by LSH banding.
- The post also argues contamination distorts checkpoint, hyperparameter and data-mix selection, because the run that memorised more of the contaminated slice comes out ahead on the score.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The first two checks cost one pass over the data with no thresholds to set, so the bill is CI minutes; the cost of skipping them is carried by whoever budgets against the reported gain.
- decision If contamination decides which checkpoint wins, the audit has to run before the selection step, not before the write-up.
- constraint A contaminated eval number cannot be discounted after the fact, so the eval set has to be re-verified every time the training data changes.
Five percent of the eval set contaminated, the model near-perfect on that slice [8]. On a hundred-point accuracy metric a 5% slice contributes at most five points, and you collect all five only if the model would have scored zero on those items unseen [21]. If it would have got half of them right anyway, the inflation is two and a half points [22]. Compare that with the size of the gain being demonstrated: the post opens on a six-point jump after a fine-tune [20].
The correction is not available afterwards. According to the post, subtracting points does not work, because you do not know how the model would have done on those items unseen [11].
The checks split by cost and by confidence [12]. Level one joins whichever fields define an example and compares byte for byte; it is a hash lookup, O(n), with nothing to tune, and it reports no false positives and no false negatives [13]. Level two applies Unicode NFKC, lowercases, strips punctuation and symbols and collapses whitespace before the same comparison. That catches an example that came back through a reformat: smart quotes, a title-cased prompt, a markdown wrapper [14]. It is still a hash lookup, and in practice it finds several times more matches than exact alone [14]. The post calls skipping it the single most common reason a contamination audit reports clean when it is not [15]. Level two costs one NFKC pass and a lowercase call.
Level three is the only check with a threshold to set. Each text becomes a set of overlapping word 5-grams, similarity is the Jaccard index of two shingle sets, and the O(n squared) all-pairs comparison is approximated with MinHash, commonly 128 permutations, grouped into LSH bands so any matching band makes a candidate pair [16]. The approximation sits in candidate generation [18]. Candidates are then scored on the shingle sets themselves, so a reported similarity is measured and not read off a signature, and very long records are first reduced to a bottom-k sketch that stays stable when the record is edited [17]. Level three can miss a small number of borderline pairs [18].
The case for CI rests on how contamination arrives. It comes in with pipeline changes: a merge of two public collections that share a seed set [2], or an augmentation step that runs before the split [3]. A June crawl of the site the eval came from in March does it [4]. So does a training file three people have appended to weekly while the eval stayed frozen [6]. A one-off audit in a notebook was true for one commit [7].
Reporting is the smaller problem. Checkpoints, hyperparameters and data mixes get chosen by comparing eval scores, and contamination rewards whichever run memorised more, usually the run that trained longer on the contaminated subset [9]. "A genuinely worse model can outscore a better one," the post says [10].
The post gives the method and the cost of running it. It does not publish numbers behind its wider claim that contamination is among the most common and least discussed reasons an offline number fails to reproduce in production [19], nor a runtime for the ten-minute check it promises [20]. In my view the first two levels are cheap enough that a fine-tuning result reported without them is unfinished work, whatever the prevalence turns out to be. The post's instruction for level one is three words: "Always run it." [13]
What to watch
- A measured contamination rate for a widely used instruction-tuning collection would test the claim that these merges are the common case.
- Whether the three-level check ships as a default step in open eval harnesses or stays a per-team script.
- Published near-duplicate similarity thresholds, which would make level-three audits comparable between teams.