A memory-mapped n-gram table still misses a quarter of page reads at 16 MB resident

The locality case rested on code being self-repetitive: an earlier experiment in the same series measured a repeat rate of 0.88 ^[4]. That figure counts tokens recurring. The table is keyed by context, and two earlier measurements had already separated those two things, finding that within one document the cache saturates at order 4 and that verbatim 5-grams never recur at any length tested ^[15]. Novel order-3 contexts keep arriving for as long as you keep typing, so each position is a fresh key into a 1.93M-context table ^[16]. The post puts it in one sentence: "the predictor is whether the context recurs, not whether the token does" ^[18].

The project that prompted the experiment is a third-party runtime that runs 35B and 80B mixture-of-experts models on Apple hardware, keeping a dense core in memory and streaming experts from SSD at reported cache hit rates of 43 to 70% ^[2]. It works because a router picks a handful of experts per layer and the same experts serve many tokens ^[17]. The n-gram table's bounded-cache hit rates sit inside that same band, 0.473 at 1 MB up to 0.744 at 16 MB ^[13][7]. What differs is where the hits come from. A dense network cannot stream at all, since every forward pass reads every weight ^[3].

The point of the rebuild was to reprice the question, from how much accuracy fits in B bytes to how many resident bytes a given accuracy costs ^[22]. The author reports the hypothesis is false ^[5]. A 143 MB table for 2 MB resident is not on offer, and at 2 MB, 43% of page reads miss ^[14]. That 2 MB is about 1.4% of the table ^[1]; reaching the top of the measured range costs 16 MB, or roughly 11% ^[2]. The return per megabyte falls off hard on the way: the three megabytes above 1 MB buy about 0.061 of hit rate each, the twelve above 4 MB about 0.0073 each ^[3].

A pageable rewrite of a lookup structure is precisely the kind of change that can be subtly wrong in a way that improves your numbers ^[23]. So each record stores its own context tokens and they are compared on every probe, which stops a 64-bit hash collision returning the wrong distribution silently ^[8]. The harness also reproduces an earlier published accuracy figure, 0.421 against 0.413 for corpus-only at L=250 ^[10]. The research repo is not public, and the numbers come from its own harnesses ^[6].

That leaves paging as an engineering trade, not a memory saving. The estimate is about 15 page reads per position, 26% of them missing at 16 MB resident, so roughly 4 faults, which at 100 microseconds a read is 0.4 ms against the completer's 10 ms budget ^[20]. Fifteen reads at 26% is 3.9 faults ^[4]. Every array in the experiment was in RAM, the page counts are analytic, and the 100 microseconds was nominal, an assumption the author then went and measured ^[21]. For the 0.4 ms to transfer to your machine, your storage has to serve a cold page in 100 microseconds at that miss rate. The budget itself breaks at about 2.6 ms a read, because 10 ms divided by 3.9 faults is 2.56 ^[5]. The write-up breaks off before the measured latency appears.