Plain-language questions push SEPA rulebook answers out of a top-5 vector search

Row 21 of the test set started as a bug report ^[5][15]. In an earlier piece, the author found a question whose answer sat at rank 7. The model only ever saw ranks 1 to 5, so it refused ^[2]. A reader, Mikhail, had found that question by hand. He then found two rewordings, both using the section's own words, that put the same passage at rank 1 ^[3]. His refused question is now the row's plain phrasing, and his working rewording is the original ^[15].

The plain column is built so the person writing the test cannot slide back into the rulebook's vocabulary ^[9]. It is the question as a customer or a new joiner would type it: "bank" for PSP, "instant transfer" for SCT Inst ^[9]. Each question has a list of forbidden terms, and the runner refuses to start if a plain phrasing contains one ^[9]. The phrasings were written before the run, and the golden set was not edited after the ranks came back ^[16]. The terse column, three to five words, may reuse section terms, because the variable there is length ^[9].

The corpus is the European Payments Council's two SEPA credit transfer rulebooks. They are cut into 484 passages of about 300 words each and embedded with OpenAI's text-embedding-3-small ^[4]. The live retriever hands the model the five passages closest by cosine similarity, and nothing else ^[6]. Five out of 484 is about 1% of the corpus ^[1]. If the passage holding the answer is not among them, the model cannot answer, however good it is ^[1]. So the score is the rank of the first labelled page among all 484, recorded once per phrasing against the stored production vectors ^[8][14]. Four questions already sat at rank 4 or 5 with their expert wording ^[10]. A judge grading final answers could pass all four ^[8]. Their ranks put them one or two slots from the cutoff ^[10].

The author tested four hypotheses. The fourth could not change anything, by construction, because every phrasing was English ^[11]. Of the rest, the author wrote: "The first held. The second and third did not." ^[12] The first predicted that plain wording would push several answers out of the top five, and terse wording more ^[11]. The second predicted that dropping duplicates would help plain and terse phrasings most, by freeing slots that duplicate passages take up ^[11]. There is a lot to drop. Of the 484 passages, 138 (about 29%) are exact duplicates, because a 36-page annex is bound into both rulebooks ^[7][2]. The third went after the hybrid path, which had cost 3 of 20 English questions in an earlier test ^[13]. The author blamed passages that matched on a single shared acronym ^[13]. The fix counted a passage only when it matched 2 or 3 distinct query words ^[14]. The excerpt ends at the results heading, so it does not include the per-question ranks or how far each answer fell.

The first result carries over to another system only if three conditions hold. Its users would have to ask questions in words the documents do not use. Its retriever would need a fixed top-k with no reranker behind it. Its embedder would need to handle that vocabulary gap about as text-embedding-3-small does. Payment rulebooks are a hard case on the first condition, since the plain phrasings had to replace terms like PSP and SCT Inst ^[9]. In my view, any golden set written by people who know the documents needs a plain column with a forbidden-term guard. Building one is cheap. The run used a read-only connection, one embedding per phrasing, and the production full-text query built on plainto_tsquery and ts_rank ^[14].