A completer that scores 0.546 on its eval scores 0.070 on the thing users see

The author of pycomplete, a local code completer, published a held-out benchmark that reads as solid: 441 files and 2,065,825 tokens of numpy indexed in 6.9 seconds, then 0.546 top-1 and 0.739 top-5 across 2,000 positions in 24 files the model had never seen, at 11.1 milliseconds a call ^[1]. He then scored the output a user actually looks at, four-token ghost text judged by exact match end to end, and it came back at 0.070 ^[2].

The parts are modest and the speed is real: a 2.48M-parameter transformer, two layers, d=128, a 256-token window, 10 MB on disk, blended with an n-gram model over the repo and a cache over the buffer above the cursor, all on a laptop CPU ^[3]. What matters is how it was tuned. The transformer mixture weight alpha=0.45, cache_beta=0.5, the order-5 count tables, and the choice of a 256-token window over 64 and 1024 each won a sweep against held-out next-token prediction ^[4]. That is a well-measured system, measured against one output.

There are two outputs. suggest() returns a single-token autocomplete list scored top-1 or top-5, which is exactly what the eval scores; suggest_multi() returns up to eight tokens of greyed-out continuation that the user accepts or rejects as one unit ^[5]. According to the author, nothing had ever scored the second one, not because it was hard but because the eval had a unit and that unit was the token ^[6]. The gap between the two numbers is a factor of 7.8 ^[1], and it is worse than a naive independence bound predicts: four tokens each hit at 0.546 would exact-match at 0.089, above the observed 0.070 ^[2].

The cause is in two lines of code. The eval feeds the model the true prefix of a real file; ghost text appends its own guess to the buffer and asks again ^[7]. One reads text a human wrote, the other reads text the model wrote, and every weight was fitted on the first ^[7].

He measured the size of that shift rather than asserting it. Replacing the last N tokens before the cursor with tokens the shipped system generated, leaving everything earlier true, and asking each component for the same true next token, at n=400 held-out positions with 64 tokens of tail: the collapse is entirely at the first step, shipped at -0.367 with 14.8 SE, and everything past f=0.25 is a plateau where the arms sit inside each other's noise ^[8]. Degradation is not gradual, so there is no gentle region in which to tune ^[9]. A re-run against a much smaller index reproduced the cliff harder, 0.435 to 0.098 across the same first step at 13.6 SE, then the same flat plateau to 0.068 ^[10]; the author notes absolute heights move with the index, so the cliff is quotable and the 4.4x multiple is not ^[11].

One row moved. The original run found the repo-counts arm essentially immune to self-generated text at -0.018, 1.1 SE, not significant; on the author's index it drops -0.103 at 5.9 SE, which he reports as unreplicated and index-dependent ^[12].

The axis label is the part worth taking away. Each component receives the whole file prefix, mean 11,441 tokens, so f=1.00 means only the 64 tokens immediately before the cursor are generated, about 0.6% of what the component can see, and f=0.25 is sixteen tokens ^[13]. Sixteen self-generated tokens at the end of an eleven-thousand-token buffer are enough to take the whole system to its floor ^[13][8].

Watch whether the four-token exact-match number becomes a tuning target, and whether alpha, cache_beta and the window size hold their values when re-swept against it. Watch, too, whether the repo-counts row moves again on a third index, since that is the one component the author has already declined to vouch for ^[12].