Build1 publisher2 min readPublished
Qwen3.5-9B's 262K context window would consume the whole 8GB budget in KV cache
A dev.to guide to running local models on 8GB prices the KV cache between 15KB and 160KB per token depending on architecture. At 32K tokens held, that spread is the difference between 0.5GB and 5GB of a fixed budget.
The Engineer · Build desk
What happened
- A dev.to guide to local models on 8GB argues that sizing fails because teams count the weights file alone, ignoring the KV cache and the 500MB to 1GB taken by the OS, desktop and inference runtime.
- It prices the cache at about 160KB per token for Granite 4.1 8B, which needs 5GB at 32K context, against about 32KB per token for Qwen3.5-9B and about 15KB for Gemma 4 E4B.
- The post attributes that gap to architecture, saying hybrid attention patterns in models like Qwen3.5-9B cut the per-token cost against older full-attention dense models.
- Gemma 4 12B QAT is presented as the largest dense model that comfortably fits, on Google's quantization-aware training builds, with an explicit caution about context length.
- For shared-memory machines such as Apple Silicon Macs the guide steers readers to Qwen3.5-4B, the Gemma 4 E2B and E4B builds, and Phi-4 Mini for math-heavy work.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Planning on 8GB starts by choosing a context ceiling and picks a model second, because the window the post recommends most warmly cannot be held resident inside the budget at all.
- decision Cache cost per token can make the larger weights file the cheaper choice. The comparison a team should run is per-token cache at its target context; parameter count will not settle it.
- exposure The failure lands at prompt time as a collapse in throughput once the runtime spills into system RAM, so an undersized budget presents as a slow model, not as a refusal to load.
- cost Holding 32K tokens is paid for out of the weights allowance: 1GB of cache plus 1GB of overhead leaves 6GB for the model, using the most cache-efficient architecture in the post.
The post calls Qwen3.5-9B "currently the gold standard" and credits its hybrid attention and 262K context [9]. At the same post's own figure of roughly 32KB per token [6], holding 262,144 tokens of cache comes to 8,388,608KB, or 8GiB [1], the whole budget with no weights resident.
Cache accumulates as context does, linearly with the number of tokens held, according to the post [4]. None of it is claimed at load time. So decide first how many tokens you intend to keep, because every one of them draws on the same 8GB.
The tightest case in the list is Gemma 4 12B QAT at 7.2GB [10]. Subtract that from 8GB and 800MB is left, and the post's own 500MB to 1GB overhead range then takes 500MB of it or all of it, leaving 300MB at best and a 200MB shortfall at worst [2].
The same subtraction on Granite 4.1 8B is harsher. Its 5GB of cache at 32K context plus 1GB of overhead leaves 2GB for weights [8]. The smallest weight figure the post publishes anywhere is 3.4GB, for Qwen3.5-4B [12].
Whether the per-token numbers transfer is a separate question. The post does not say how they were measured or at what precision the cache is stored [15]. They also multiply out to round totals: 32,768 tokens at 160KB is exactly 5GiB, at 32KB exactly 1GiB, at 15KB 0.47GiB [3]. Those look like targets picked first with per-token rates fitted to them afterwards. For the ratio between two models to hold on your hardware, your runtime has to keep the cache in the same format the author assumed, and you have to actually fill the context [4].
The route past the ceiling in the post is Mixture-of-Experts offloading: Unsloth dynamic quantization that keeps the essential layers on the GPU and pushes rarely used expert tensors to system RAM, invoked as `llama-server -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q2_K_XL -ngl 999 --n-cpu-moe 30` [13]. The post warns that the flag moves the bottleneck from VRAM capacity to system memory bandwidth [14]. It also sets a floor of 32GB of system RAM, four times the ceiling the rest of the article is written around [5].
What to watch
- An independent measurement of resident cache at a full 32K context on Granite 4.1 8B and Qwen3.5-9B would either confirm the 160KB and 32KB per-token figures or move them.
- Published weight sizes for the 8B and 9B builds recommended here, since the residual cache budget depends on them.
- A per-token cache figure for Gemma 4 12B QAT, the 7.2GB build with the least headroom on the list.