Enriching 1,245 tables with an LLM pushed the word contact into 1,072 documents

The 0.15 checks out. The post gives BM25's inverse document frequency as idf(t) = log(1 + (N - n_t + 0.5) / (n_t + 0.5)) ^[4]. Put N at 1,245 and n_t at 1,072: the fraction is 173.5 / 1,072.5, and the natural log of 1.1618 is 0.150 ^[16]. That puts the query term in about 86 percent of the corpus ^[15]. The tokenizer in that index does not strip stopwords, so "the" and "and" are in there too, scoring near zero ^[6].

The model is describing each table correctly. Asked to describe a CRM table, it writes about contacts when describing `contacts`, and again when describing `contact_lists`, `campaign_recipients`, `email_events`, `tenants`, `users`, and the audit table that logs changes to any of them, because in a CRM almost everything is about contacts in some defensible sense ^[20].

The second effect comes from the length prior. BM25 divides the term-frequency contribution by k1 * (1 - b + b * len / avg_len), penalising long documents on the theory that a long document containing your term is less about your term ^[7]. The `contacts` table has 55 columns, and with a generated description and a row of alias words its document runs several times the corpus average ^[8]. `contact_import_log` has six columns and a one-line description ^[9], roughly a ninth of the column count ^[17]. The post's claim is that the most important table in a schema is reliably the one with the most columns, so once IDF has flattened, the length prior sorts the remainder by inverse centrality ^[19].

Cataloguing did not add noise, the author wrote; it "added correlated noise, and correlated noise attacks the exact query it was meant to help" ^[10]. The degradation profile follows from that. Questions that named a table outright were mostly fine, because they carried a rare token; the questions that fell apart were plain English with no schema words ^[11]. That is the population cataloguing was bought for.

Down-weighting the generated text is the first thing most teams will reach for, and the author reached for it first. It helped a bit and the post calls it the wrong lever: weight and dilution act at different stages, because down-weighting scales a term's contribution after IDF has already been computed over the corpus the descriptions polluted ^[12]. If you want the separation back, you have to change what the statistics are computed over.

Scope matters here. The author says the same failure is available to any pre-index enrichment: summaries, generated titles, hypothetical questions, keyword expansion ^[13]. What is actually measured is a lexical index, and the diagnosis is stated entirely in BM25's IDF and length-normalisation terms ^[18]. A claim about dense retrieval collapsing the same way would need its own numbers, from an index where document frequency does not enter the score at all. The supplied text of the post ends mid-sentence, before it sets out the fix the author says does work ^[14].