Build1 publisher3 min readPublished
Every unanswerable question cleared the 0.35 refusal threshold by at least 0.09
A Java engineer measured his own retrieval pipeline and found that the 0.35 similarity floor had never rejected anything. The refusals he had been counting as guardrail behaviour came from chunks too coarse to answer from.
The Engineer · Build desk
What happened
- A Java backend engineer built his retrieval-augmented generation pipeline on the JVM, using Java 21, Spring Boot, Spring AI and PostgreSQL with pgvector instead of the Python and LangChain stack the tutorials used.
- Before adding features he built an evaluation harness: 20 fixed questions against one ingested document, his own CV, with 12 answerable from a single passage, 5 needing two or more, and 3 the document does not contain.
- Cutting chunk size to 150 raised retrieval accuracy to 76% and correctness to 59%, and dropped correct refusals to two of three.
- The per-question scores showed the three unanswerable questions matching their best chunk at 0.440, 0.463 and 0.464, all of them above the configured threshold.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint One cosine cutoff cannot sort these two classes on this document. A floor high enough to reject every unanswerable question sits above scores that real answers also produce, so the repair costs correct retrievals.
- exposure Finer chunks hand the model a specific nearby number to attach to the wrong project, so the wrong answers the pipeline now produces are harder for a reviewer to spot than the vague ones it used to give.
- decision Anyone tuning against the refusal column is tuning against three data points, where a single question swings the figure by 33 points; the set has to grow before that column can carry a decision.
- precedent A threshold nobody had measured behaved exactly like no threshold at all. Nobody had set the 0.35 line beside an observed score distribution.
In run 1 the component declining unanswerable questions was the chunker. The CV runs a few hundred words, Spring AI's default chunk size is 800, and the whole document came out as two chunks [9]. Every query, however specific, was embedded and scored by cosine similarity against those same two blocks [3][9]. The retrieved context was too general for the model to build an answer from, so it declined [14]. The gate at 0.35 had no part in that. The lowest similarity among the three unanswerable questions, 0.440, cleared the threshold by 0.090 [5].
"The threshold had always been completely inactive," the author wrote in a post on dev.to [12]. The pipeline had been retrieving real context for questions the document cannot answer, then relying on the model to volunteer that it did not know [13]. He ranks refusal above the other two criteria: "a system which confidently answers a question for which it has no basis is worse than one which returns no answer" [18].
At chunk size 150 the sections became thematically distinct and the accidental refusal went away [14]. Asked for the load-test response time on one project, a figure that does not apply to that project, the pipeline took the 2-3 ms value from a similar project in a nearby section and assigned it to the wrong one [15]. "Better retrieval made the hallucination seem more believable," the author wrote [16].
Raising the threshold is the obvious repair, and the score distribution blocks it. Ranked by top-chunk similarity, the highest unanswerable question scored 0.464 and the answerable band starts at 0.423 [17]. A cutoff placed above 0.464 to catch all three unanswerable questions therefore also rejects answerable ones scoring between 0.423 and 0.464 [6].
The movement in the headline table is smaller than the percentages make it look. Seventeen of the 20 questions are answerable, and that is the n in both accuracy columns [1]. Retrieval going from 59% to 76% on 17 questions is about three questions [2]. Correctness from 47% to 59% is two [3]. The refusal column has three items, so falling from three to two is one question, worth 33 points [4].
For that table to say anything about a different pipeline, the corpus would have to behave like this one: a single short document, split into two blocks by the framework default, with scores from the same embedding model. The post does not name the embedding model [19]. The check is the part that transfers. It is the top-chunk similarity logged for every question, including the ones expected to be refused, set beside the number in the config. In the run whose per-question scores the post publishes, no question scored below 0.423, so the 0.35 line rejected nothing [7].
What to watch
- Whether a larger unanswerable set than three questions preserves the 0.423 to 0.464 overlap or widens it.
- Whether a different embedding model separates answerable from unanswerable where top-chunk cosine does not.
- Whether the author adds a second check, such as reranking or an answerability classifier, and reruns the same 20 questions.