Sixteen times more training code buys the same gain as reading the rest of the file

The vocabulary is where this sweep could have broken without anyone noticing. Each checkpoint built its own 16K vocabulary from its own subset of code, and only 39.8% of the 500K and 8M vocabularies overlap ^[5]. So every cell is scored in the vocabulary of the checkpoint under test, and the corpus count model in that cell is built from the same tokens the transformer saw in training ^[5]. Nothing travels between rows except the position of the crossover ^[6].

The cache row is the control. It never touches the training corpus, so it should be flat, and it is: top-1 of 0.311, 0.312 and 0.313 at 250 tokens of document, and 0.487, 0.488 and 0.487 at 4,000 ^[11][4].

The fit is checkable. Sixteen times is four doublings, and 1.60 to the fourth is 6.55 ^[23]. Fit only the 2M-to-8M segment, where the model is less undertrained, and the rate rises to 1.78 per doubling. At that rate a 4,000-token file needs about 97M training tokens, some 12 times the 8M corpus ^[13][24]. The post calls that extrapolation illustrative rather than measured, and cites the project's own earlier work as a warning against trusting a slope past its fitted range ^[14].

The crossover moves because the trained line rises, and it rises by about the same amount at every document length. The gain from the full 16x is +0.132 at 250 tokens and +0.148 at 1,000, barely different from the figure at 4,000 ^[16]. The window is 64 tokens, and no quantity of training text widens it ^[16]. Spread over four doublings, the 4,000-token gain works out at roughly 0.043 of top-1 per doubling of the corpus ^[28].

The transformer wins exactly one cell in the table, at 8M tokens and 250 tokens of document ^[9]. In that same cell the corpus count table scores 0.415 against the transformer's 0.370, a margin of 0.045 for counting ^[19][25]. A neighbouring experiment's corpus-plus-cache combination runs from 0.461 to 0.551 across these lengths ^[20].

For this exchange rate to transfer, several things have to hold: in-distribution held-out code ^[10], a model of about 2.45M parameters with a 64-token window working on files of thousands of tokens ^[21][16], and a setting where the extra 10 to 20 times in-domain code cannot be obtained ^[15]. Per-repo completion is that setting, according to the post ^[29]. The repo is not public, so every number here comes from the project's own harness ^[2]. The 8M, 250-token result does reproduce an earlier finding that counting beats the small transformer on code, on a different harness ^[19].