Build1 publisher3 min readPublished
Fusion of body, vector and prose signals, not just the word "refunds," carries billing_credit_note into the top six
Name-only BM25 matched nothing for "money we gave back to shoppers". The description channel put the right table second and embeddings put it twenty-second, and reciprocal rank fusion let the blind channel cost the answer nothing.
The Engineer · Build desk

What happened
- Asked "money we gave back to shoppers" against a 42-table schema, BM25 over table and column names alone matched nothing, because none of those four words appears in any identifier.
- BM25 over the generated descriptions ranked the answer, billing_credit_note, second, on the word "refunds" in its description, while embedding similarity ranked the same table twenty-second.
- Reciprocal Rank Fusion with K equals 60 summed the four channels to 0.04472 and carried the table into the top six, with the names channel contributing 0.00000.
- On the same 52-question fixtures, run by someone else from the published wheel, identifiers alone answered 29, one flat bag answered 41, prose as its own channel answered 48, and the shipped configuration answered 49.
- At 1,245 objects, descriptions from one model put the token "contact" into roughly 1,072 documents, and the table called contacts fell from rank 3 to below rank 40 on a question using that word.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- contradiction The configuration that wins this question is the one that broke at scale: one bag over names, columns and descriptions ranked the answer first on 42 tables and collapsed on 1,245 objects.
- constraint Because a cosine distance and a BM25 score are different units, teams adding a fourth channel have to fuse on positions, and accept that positions weight a lucky near-miss almost as heavily as a correct top hit.
- cost Rank fusion makes a meaningless ranking free to cast a full vote, so a filler word shared by every description hands the tie-break to the order the catalog happened to be built in.
- decision Anyone generating table descriptions with a single model now has a corpus to police, not just sentences to check for accuracy, since a correct word repeated across most objects destroys its own discriminating power.
Rank fusion buys the fourth channel cheaply because K is large. First place pays 1/61, which is 0.01639. Twenty-second pays 1/82, or 0.01220 [13][14]. A hit at rank 22 is therefore worth about 74 percent of a hit at rank 1 [15]. The embedding channel put twenty-one tables ahead of the right one and still supplied roughly 27 percent of the winning total [16][8]. Between the body channel at first and the prose channel at second, the difference was 0.00026 [17].
What the prose channel matched on is one generated sentence. The description reads "Records refunds or corrections issued against a specific invoice." [6] "refunds" is the term that links "gave back" to a credit note, and it appears nowhere in the DDL [7]. The names channel, which reaches 100 percent recall@6 when people phrase questions in the schema's own vocabulary, matched nothing at all here [4][3]. The post also notes there is no per-term score to inspect when a dense ranking comes back slightly off [9].
Those four rankings are one question on a 42-table schema, measured on a single run [2][33]. The wider evidence is a 52-question fixture set, and the author says he got its central question wrong first, at the cost of a public retraction [21]. One flat bag over names, columns, types, comments and descriptions answered 41 of 52, or 78.8 percent; giving prose its own index answered 48, or 92.3 percent [22][23]. That is a gap of seven questions [24]. Inside one bag the sentences compete with the identifiers for term frequency, and the identifiers lose, because a description has more words in it than a name does [25].
In the names index, "contact" appears in 17 of 1,245 objects, and its IDF is 4.27 [27]. Across the generated descriptions it appears in roughly 1,072 of 1,245, about 86 percent of the corpus, and IDF falls to 0.15 [26][28][27]. An accurate English word, generated into 86 percent of the documents, then scores like "the". The post says the descriptions were accurate and that their accuracy does not save them, because IDF is computed over the corpus and not over the sentence [30].
The cost of ranking-based fusion lands on tie-breaks. A channel that ordered its results for no meaningful reason still votes at full strength, so if every description contains the word "about", every object ties in the prose channel and the order the catalog was built in decides the ranking [18]. Shuffling insertion order changed which tables came back on 1 to 2 questions out of 12, roughly 8 to 17 percent, without moving recall [19][32]. A commenter on the author's previous post raised this, and the author says both fixes proposed are right and both are filed [20].
For the shipped 94.2 percent to transfer, two conditions have to hold in your catalog. Users have to ask in business vocabulary, because on schema-vocabulary questions the names channel already returns the answer in the top six on its own [4][22]. And the generated descriptions have to avoid putting one common word into most of the corpus, which is what happened at 1,245 objects [26]. The post credits fielded scoring for the fix, and says the reason is structural, not a tuning choice [31].
What to watch
- Whether the two filed fixes for full-strength voting from a tied channel move recall, or only stabilise what lands in the prompt.
- A per-channel recall figure on the 1,245-object schema, which would show whether fielded scoring holds at that size or only relieves the worst token.
- Whether descriptions generated by more than one model, or written against an enforced vocabulary, keep the IDF of words like contact.