Similarity is the only thing the pipeline measures. A question about leave entitlement matches the 2023 policy about as well as it matches the 2026 one, because both documents are about leave entitlement; nothing in either marks which is live [5], and a build that stuffs the top matches into a prompt has nowhere else to look [2]. The three years between the versions do not help [1]. The slide deck made for one client has the same problem in reverse: it reads like an official company position because that is how it was written [6]. Retrieval returns what it finds, faithfully [7].
The part worth arguing with is the claim that the model is the least interesting component [3]. The danger the post names is a model behaviour. A list of search hits lets a reader notice that the top result is a draft out of somebody's personal folder; a fluent summary of that same draft arrives in the identical register as a correct answer and hides the document underneath it [8]. The model did not create the contradiction. It removed the review step that used to catch it, which makes fluency an argument for curation rather than a replacement for it.
The two rejected access shortcuts fail at the same boundary. Filtering results after the fact and instructing the model to keep certain things secret both admit restricted passages into the context and then try to manage what happens next [17][18]. Pushing the check into the query instead means the index has to carry access rules from the source system, and group membership has to stay in sync as people change roles [19]. Curation gets described as the single highest-leverage day on the project [10], and the list it produces is three fields wide per topic [2], but nothing about that list stays true on its own. Document status changes and staff move teams without the index being told.
The clearest sign this is a governance job rather than an engineering one is the by-product. Teams doing the exercise routinely find that two departments have been operating on different rules for a year, which the author rates as worth more than the chatbot [11]. No retriever produces that finding. It comes from a person with authority reading the same procedure written three times by three departments that quietly disagree [6]. On the intake systems the author builds, including a voice and SMS product for Community Action Agencies answering in more than 100 languages, the work starts with eligibility rules and program policy rather than speech, and ambiguous policy is not rescued by a better model [15]. Engineers dislike this stage because there is no library for it [12], which is the honest reason it keeps getting deferred until it reappears as a retrieval bug.
