DeepSeek paid for Engram's lookup table by deleting 17 of its 72 routed experts

Twenty-one percent of the parameter budget ended up in the lookup table, and it was funded by subtraction ^[1][12]. The 17 experts DeepSeek dropped are 24 percent of its routed pool ^[2], and the 5.7 billion parameters that took their place amount to roughly 1.5 times the model's entire activated parameter count ^[3]. The comparison, as relayed in the Hugging Face explainer, was built to hold total parameters and per-token computation constant ^[13], so what is on offer is a result about allocation at one budget, not evidence that a new axis keeps paying as you push it.

The design leans on defensive tricks because of the counting. Blackroot's illustrative 8,000-token vocabulary already yields 64 million possible bigrams ^[7], and the same vocabulary yields about 512 billion possible trigrams ^[6]. Set that against 5.7 billion parameters of memory and you get at least 90 candidate trigram keys per parameter ^[7]; because each table entry is a vector of many parameters, the true oversubscription is worse, and modern vocabularies are larger than 8,000 ^[7]. Fixed-size tables and accepted collisions follow from that arithmetic rather than from convenience ^[8]. The multiple hash functions, tokenizer normalisation and context-aware gate are load-bearing components of the mechanism, not polish ^[8].

The gate is the honest part. Local sequences sit on a steep frequency curve, so common phrases get enough examples to learn a useful vector while rare strings land on undertrained entries or collide with something unrelated ^[9]. The gate lets the model discount a retrieved vector when the surrounding context suggests the lookup is not helping ^[9]. In other words, a chunk of the parameter budget is spent on capacity the model must also learn to distrust.

"Cheap" is carrying weight in the pitch ^[18]. A lookup replaces arithmetic, not residency: the retrieved vectors are learned during training and stay part of the model's parameters ^[5], so the saving lands in compute per token while the server still has to hold all 26.7 billion ^[11]. That is the same bargain mixture-of-experts already makes with conditional computation, applied to conditional lookup instead ^[10]. Worth noting how small the evidence base is: the whole comparison ran on 262 billion training tokens ^[11], under 10 tokens per total parameter ^[5], which is a thin run on which to settle how much static memory beats an expert.

The explainer arrived more than seven months after the January 12th paper, with no release or confirmed production deployment attached ^[17][14]. What gives it currency is the audience: Blackroot's Hugging Face profile hosts dozens of uploads including quantised and modified language models ^[3], which is exactly the constituency that has to decide where parameters go rather than admire that they exist. Blackroot's compression of the idea, "Engrams are embeddings. With extra steps." ^[2], is closer to the engineering than the scaling-law framing is. Liang Wenfeng is one of 14 authors on the paper ^[14] and told the Associated Press that his days go to reading papers, writing code and joining research discussions ^[15], so the budgeting question came out of the lab, not a product plan.