A 284B model at 25 tokens a second on one 5090, and 192 GiB of DDR5 doing the quiet part
FreeToken's preprint puts DeepSeek-V4-Flash on a single RTX 5090 desktop. The GPU holds about a sixth of the machine's memory, and nobody outside the team has run the benchmark yet.
The Engineer · Build desk
What happened
- A preprint on the FreeToken engine reports 22 to 25 decode tokens per second for the 284-billion-parameter DeepSeek-V4-Flash on a single RTX 5090 desktop.
- Table 1 lists that desktop as a Ryzen 9 9950X3D with 192 GiB of DDR5 beside the card's 32 GB of VRAM, all treated as one serving platform.
- The same paper reports 39.3 tokens per second for a quantised Qwen3.6-35B-A3B on an 8 GB RTX 4060 laptop under a coding workload.
- The benchmarks have not been independently replicated, and the paper was submitted on August 17, 2026.
Why it matters
- cost The bill for reproducing this is not one GPU. It is a top-end desktop CPU and 192 GiB of DDR5, which is where a lot of buyers will find the number moves.
- capability If it survives outside contact, work that was scheduled onto cluster time becomes a thing one engineer runs at their desk, at a pace closer to reading than to chat.
- constraint A decode-only metric leaves the part agents pay for most, long contexts being processed again after every edit, outside the reported result.
- exposure The llama.cpp, Ollama and KTransformers comparisons stand or fall on the authors' own test setup, which puts the ranking, not just the headline, up for challenge.
The sparsity is what makes the claim plausible before anyone checks it. DeepSeek-V4-Flash routes each token through six of 256 experts in each of 43 layers, with roughly 13 billion of 284 billion parameters active [10]. That is about 4.6 percent of the weights per token [19] and about 2.3 percent of the experts in any given layer [21]. The paper says the active computation fits inside the 5090's 32 GB of VRAM while the full expert pool stays much larger [11], so the hard part is not arithmetic. It is fetching: which experts sit in VRAM, and what to do when the router asks for one that does not.
FreeToken's answer is a global least-recently-used expert cache, with cache misses split between PCIe transfer and CPU execution according to the bandwidth available at that moment [13][12]. Prompt processing uses full-layer double buffering, staging the next layer while the GPU works the current one [13].
Which is why Table 1 is the more informative page. Alongside the 32 GB card sit 192 GiB of DDR5 and a Ryzen 9 9950X3D [9], roughly six times as much memory outside the GPU as inside it [20]. The GLM-5.2 figure makes the same point from the other direction: 14.9 tokens per second on a single 96 GB RTX PRO 6000 [8], a card with three times the 5090's VRAM, on a different model. Card capacity is not the variable this work is reporting.
Then the metric. The paper labels it decode throughput, per-request mean tokens per second, and does not state elapsed time from prompt submission through completion [2]. At 22 to 25 tokens per second, a 1,000-token reply is 40 to 45 seconds of decode alone [18], before prefill. The project's own agent machinery suggests prefill is not cheap: semantic-aware checkpointing at thinking segments, tool calls and conversation turns exists so the engine can re-prefill only a new suffix after an edit [15], and the GPU expert cache can be rebuilt without reloading the host-resident pool [16]. Features built to avoid re-prefilling concede that prefill costs something the headline number omits.
The comparison set, six GPU systems over four workloads against llama.cpp, Ollama and KTransformers in sections 5.1 and 5.2, was run by the authors [4], and the benchmarks have not been independently replicated [14]. The roster, Berkeley's Sky Computing Lab plus Keutzer, Han, Zaharia and Stoica [17], is why the preprint will be read this week. It is not the same thing as a second machine producing the same tokens per second.
What to watch
- An independent run of DeepSeek-V4-Flash on a 5090 with less than 192 GiB of system memory, showing how much of the result belongs to the host.
- Publication of end-to-end latency including prefill, rather than per-request mean decode throughput.
- Responses from llama.cpp, Ollama or KTransformers maintainers on the comparison conditions used in sections 5.1 and 5.2.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence44
- Adoption12
- Hype gap+16
- Incentives58
- Confidence52
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
The paper reports 22 to 25 decode tokens per second for DeepSeek-V4-Flash on an RTX 5090 desktop, with the system using CPU compute and 192 GiB of system memory alongside the GPU.
- [2]
The paper labels the metric "decode throughput (per-request mean tok/s)," a measurement that does not specify total elapsed time from prompt submission through completion.
- [3]
FreeToken still relies on substantial CPU and system-memory resources, including the host's memory capacity, memory bandwidth and PCIe connection.
- [4]
Sections 5.1 and 5.2 of the paper evaluate six GPU systems across four workloads, with comparisons against llama.cpp, Ollama and KTransformers.
- [5]
Shuo Yang and Xiaoze Fan are equal-contribution first authors of a paper on FreeToken, an open-source inference engine that pools CPU, system memory and GPU resources to run Mixture-of-Experts models on local hardware.
- [6]
The preprint was submitted on August 17, 2026.
- [7]
The paper reports an NVFP4 version of Qwen3.6-35B-A3B at 39.3 tokens per second on an RTX 4060 laptop GPU with 8 GB of VRAM under a coding workload, on a machine with an Intel Core i9-13900H and 32 GiB of LPDDR5.
- [8]
The paper reports GLM-5.2 at 14.9 tokens per second on one 96 GB RTX PRO 6000.
- [9]
Table 1 of the paper lists the RTX 5090 desktop as an AMD Ryzen 9 9950X3D processor, 192 GiB of DDR5 system memory and a GPU with 32 GB of VRAM; the project treats VRAM, CPU compute, system memory, PCIe bandwidth and local storage as a combined serving platform.
- [10]
DeepSeek-V4-Flash is a sparse Mixture-of-Experts model from DeepSeek-AI with 284 billion total parameters and roughly 13 billion active parameters per token, routing each token through six of 256 experts in each of 43 layers.
- [11]
The paper says the active computation can fit within the RTX 5090's 32 GB of VRAM while the complete expert pool remains much larger.
- [12]
The paper describes CPU-GPU co-execution, expert caching and bandwidth-adaptive scheduling, with the scheduler assigning work according to available CPU and GPU bandwidth when an expert is absent from the GPU cache.
- [13]
During prompt processing the engine uses full-layer double buffering, transferring data for an upcoming layer while the GPU works on the current one; for token generation a bandwidth-adaptive policy assigns cache misses between PCIe transfers and CPU execution, and a global least-recently-used cache keeps recently selected experts in VRAM.
- [14]
FreeToken's benchmarks have not been independently replicated.
- [15]
The paper describes semantic-aware recurrent-state checkpointing for agent sessions, placing checkpoints at thinking segments, tool calls and outputs, and conversation turns, so the engine can re-prefill only a new suffix after an edit; it cites OpenClaw, OpenCode and SWE-agent as workloads where that state reuse can matter.
- [16]
The authors say FreeToken can rebuild its GPU expert cache without restarting the engine or reloading the host-resident expert pool.
- [17]
The paper comes from researchers affiliated with UC Berkeley and other institutions; Yang is a UC Berkeley EECS Ph.D. student advised by Ion Stoica, Fan is a Shanghai Jiao Tong University undergraduate and visiting researcher at Berkeley's Sky Computing Lab, and the author list includes Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu and Stoica.
- [18]
At the reported 22 to 25 tokens per second, a 1,000-token reply takes 40 to 45 seconds of decode time.
- [19]
Roughly 4.6 percent of DeepSeek-V4-Flash's parameters are active per token.
- [20]
The RTX 5090 test machine carries about six times as much system memory as GPU VRAM.
- [21]
About 2.3 percent of a layer's experts are activated for any given token.
- [22]
The results suggest MoE scheduling can move some large-model inference from GPU clusters onto high-end PCs.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- runtimewire.comFreeToken reports 22-25 tok/s for a 284B DeepSeek model on one RTX 5090
1 article · August 22, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Benchmark Methodology and ReproducibilityFollow
- CPU-GPU Offload SchedulingFollow
- Agent Runtime State ReuseFollow
- Mixture-of-Experts ServingFollow
- Local LLM InferenceFollow
Entities
- FreeTokenFollow
- deepseek-v4-flashFollow
- Qwen3.6 35B-A3BFollow
- GLM-5.2Follow
- llama.cppFollow
- OllamaFollow
- KTransformersFollow
- MoE-InfinityFollow
- Shuo YangFollow
- Xiaoze FanFollow
- Ion StoicaFollow
- Matei ZahariaFollow
- Kurt KeutzerFollow
- Song HanFollow
- UC Berkeley Sky Computing LabFollow
- DeepSeek-AIFollow
- SWE-BenchFollow
- OpenClawFollow
- OpenCodeFollow
- SWE-agentFollow