Build1 publisher3 min readPublished
A bare except handler hid thousands of unreadable tables behind an 86.1 percent recall score
The loader read a Spider 2.0 checkout with rglob and swallowed every read error, so 2,868 files Windows would not open became tables that do not exist, and the benchmark scored recall against what was left.
The Engineer · Build desk

What happened
- A stranger spent four days reproducing a developer's retrieval benchmarks and found six defects, every one of which had been reporting success.
- The run reported 86.1 percent recall at k=20 against a database the post says was missing roughly 39 percent of its tables.
- The shortfall tracked where the repo was cloned: a 182-character root left 2,868 files unreadable and a 183-character root left 3,056.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Swallowing read errors makes a recall score partly a measure of how much of the corpus the filesystem allowed through, so 86.1 percent cannot be compared against a run on another machine.
- exposure Consumers of Scored.reason were told to branch on five values while the selector emits six, so their code takes the wrong path once in every six results without raising anything.
- precedent One character of directory name changed the corpus by 188 files, so two reviewers running the same command on the same commit can publish different recall figures and each be describing their own path layout.
2,868 of 7,892 is 36.3 percent [8]. The post puts the missing share at roughly 39 percent [6]. The larger count fits better: 3,056 of 7,892 is 38.7 percent, and 3,056 is what the post reports for a 183-character checkout root, against 2,868 for a 182-character one [7][8]. One extra character in a directory name cost another 188 table files [9]. The loader walked the checkout with `path.rglob("*.json")` and wrapped each read in `except Exception: continue` [2]. `rglob` returns the path. `open` refuses it, because Windows will not open a path over 260 characters [3]. What comes back is `FileNotFoundError` for a file that is sitting on disk, and the handler counted every one of those tables as a table that does not exist [5]. "A crash is loud. A harness that prints 86.1% when it measured the wrong thing is not, and I published every one of those numbers," the author wrote [23]. The held-out evaluation is the more instructive failure. Six fixtures get built. One of them declares three extra schemas with four objects in them: three tables all named `account`, plus a view joining across two, and that collision is most of what the fixture exists to test [16]. The harness opened a bare SQLite connection with no `ATTACH`, so `executescript` raised `unknown database "billing"` on the first of those statements. The run died there, after the five per-schema rows had printed and before the TUNE, HELD OUT and OVERALL lines [17]. According to the post, that file had never printed an aggregate at all [18]. `ATTACH DATABASE ':memory:'` before the DDL clears the exception: the script completes, the harness prints its aggregate, exit code 0 [19]. In-memory databases are dropped when the connection closes, and the harness closes that connection before `bootstrap` opens a new one [20]. The aggregate then reports 256 objects indexed, all in `main`, and no objects named `account` [20]. "The repair converted a loud failure into a quiet one, and both of us verified the repair by reading a number that the repair had not changed," the author wrote [21]. The arrangement that works is file-backed databases attached on a connect listener, with the schema list passed to bootstrap [22]. One check reported success while being incapable of failing. The reviewer reported that 52 of 52 questions had a step where the returned-set size decreases as K grows, and the author's check reported 0 of 52 [13]. That check compared `returned(K)` against `returned(K-1)`, and since top-K picks are nested and both expansion passes are unions, `returned` cannot decrease in K, so the test could never fire [14]. Comparing the overages instead of the totals gave 52 of 52, matching the reviewer [15]. One of the six defects is documentation. `Scored.reason` documented five values, `hybrid | vector | lexical | fk | pinned`, and the selector writes a sixth, `covers`, so a caller branching on the field falls through silently on one case in six [12]. Forty lines above the block that does the displacing sits the comment "Bounded, and only ever additive -- it cannot displace a ranked pick". The code had been displacing a ranked pick since a commit made a day and a half earlier [10][11]. The comment shipped in three releases [11]. This is one developer's account of his own harness, published on dev.to, and nobody in the record has reproduced the counts independently [25]. For the 2,868 figure to say anything about another benchmark, three conditions have to hold: the run executes on Windows, where `open` refuses paths over 260 characters [3]; the loader enumerates the corpus and swallows read errors instead of counting them [2]; and the corpus has paths deep enough to cross the limit from wherever it was cloned [7]. Spider 2.0's checkout meets the third at a 182-character root [7].
What to watch
- A rerun of the ablation over a full 7,892-file checkout would show how much of the 86.1 percent was the truncation.
- Whether the documented value list for Scored.reason gains covers, since callers are expected to branch on that field.
- Whether the file-backed connect-listener fix makes the fixture index its three same-named tables and the cross-schema view.