A 284B model in 3.2GB of RAM turns sparse MoE into a disk bandwidth problem

A developer has published a plain-C99 inference engine that runs a 284-billion-parameter checkpoint, roughly 160GB on disk, on a laptop with less than 32GB of RAM, at a peak resident footprint of 3.23GB, with no quantization and no rented multi-GPU box ^[1][2][4]. The stunt is not the point. The point is that for a sparse model, the binding constraint stops being memory you buy once and becomes bandwidth you pay for on every token. The mechanism is routing. According to the post, the model has 43 layers of 256 experts each, and only the top 6 experts per layer fire for any given token ^[5]. That is 258 of 11,008 expert slots, about 2.34 percent ^[1]. So the engine does not load the model; it streams it, with the router deciding which experts a token needs and the loader pulling only those off NVMe into an LRU cache ^[7]. The 3.23GB resident set is about 2 percent of the 160GB on disk ^[2]. RAM becomes a throughput dial rather than an entry fee: more of it grows the cache and raises the hit rate, and almost none still produces correct output, just with more re-reading ^[7]. The bandwidth bill is visible in the author's own numbers. Expert reads are 12.75MB each, roughly 5,000 of them per run, and the read itself was measured at 2.81ms ^[13][15]. That is about 4.5GB/s of effective throughput per read ^[3]. Cross-check it against the routing: 258 experts at 12.75MB is 3.29GB per token, which is essentially the "about 3.2GB per forward pass" the author quotes ^[4][6]. At 2.81ms per expert, pure read time is roughly 0.73 seconds per token, or about 44 percent of the 1.65s midpoint of the reported 1.6 to 1.7 seconds per token, which is around 0.6 tokens per second ^[5][3][9]. Nearly half the wall clock is the storage device. The whole run moves about 63.75GB off disk, roughly 0.4 times the checkpoint, for what works out to something like 19 tokens of output if every routed expert is a miss ^[6][7]. That reframes the purchasing decision. The question is no longer whether the weights fit, but how many GB/s the drive sustains and what fraction of routed experts the cache absorbs. Two pieces of the engineering deserve attention from anyone tempted to trust homebrew kernels. First, the author is blunt that fluent output is not evidence of a correct implementation, since a swapped nibble or misindexed scale factor still reads fine ^[9]. The stated verification is 14 kernels checked individually to 5e-7, three layer types across 46 positions at the same tolerance, and an end-to-end tiny checkpoint matching PyTorch to 2.9e-6 with identical argmax at every position ^[10], against a PyTorch reference written from scratch rather than derived from the C ^[11]. The scalar, OpenMP and AVX2 paths are held to bit-identical output using a fixed 16-accumulator reduction tree and -ffp-contract=off; the GPU is held only to relative error plus argmax match, with the CPU as ground truth ^[12]. Second, the cost the author almost missed. O_DIRECT demands 4096-byte alignment, so the first version widened each read and memcpy'd the payload out of a staging buffer, a copy assumed to cost about 1ms and never measured ^[13][14]. Measured, it was 3.60ms per expert, which the author puts at 22 percent of the cost ^[15]. Those figures do not reconcile against a single 2.81ms read, which would make the copy over half ^[8], so read the per-expert accounting with care. The alignment-residue fix that removed the copy was wrong on the first attempt, letting one tensor's widened read reach up to 4095 bytes backward into the previous tensor, and every existing test passed ^[16][17]. Watch two things.