A semantic cache shipped on by default at 0.92, and the label error that moved its own numbers

Where the embedder fails matters more than the fact that it fails. Negation has been a known weakness of embedding models for years ^[6]. Put that weakness in a reranker and you get a worse-ordered list, with the answer still in front of a user who can judge it. Put it in a cache lookup and generation is suppressed: the stored answer for the opposite rule comes back at full confidence, with nothing in the output to say a substitution happened ^[7]. Since the product's whole format is a summary plus the document it came from ^[2], a false accept does not look broken. It looks sourced.

The shipped default was wrong in both directions at once. At 0.92 ^[3], the threshold sat 0.0784 below the worst adversarial pair the author measured and 0.1730 above his weakest genuine paraphrase ^[1], which means it was simultaneously admitting a pair with opposite correct answers and rejecting paraphrases the cache existed to catch ^[10]. Raising the number does not rescue it, because the band the fix would have to live in is empty ^[11].

Then the measurement itself came apart, and this is the part worth keeping. Two of the five negation pairs were negative interrogatives, which in Spanish ask for confirmation of the same fact rather than its opposite, so a correct system answers both identically and the cache should accept them ^[14]. The author had them labelled as must-rejects. The published minimum for that row, 0.9702, turned out to belong to one of the invalid pairs ^[16], and the headline fraction of adversarial pairs above threshold moved from 60 percent to roughly 54 percent ^[2]. No test suite would have caught this, because the error was in the linguistics, not the code ^[15].

That is the argument against hit rate as the metric of record. A cache can count its own hits with no labels at all, which is why hit rate is the number that ships. A false-accept rate cannot be computed without someone deciding, pair by pair, whether two questions mean the same thing, and the person best placed to do that decided wrong on two of five pairs in a corpus he wrote himself. None of the corrections changed a single response the system returned. The served answers were identical before and after the erratum; only the claim about their correctness moved.

The honest limit is that the pairs were hand-built rather than drawn from a query log ^[9], so nothing here says how often the collision fires in production. It says something stronger about defaults: there is no setting of this component that makes the question unnecessary to ask. The reviewer's challenge took six days to answer, with the commits running 6 to 11 August ^[12], and the conclusion only survived because the 0.9984 pair happened to be a valid one ^[17]. A component that suppresses the model, shipped with no flag ^[3], puts that kind of luck between the user and the rule.