Build1 publisher2 min readPublished
A 4-bit 7B model that fits in 4 GB still runs out of memory near 30,000 tokens
Quantized to about 4 GB, a 7B model still runs out of GPU memory near 30,000 tokens as its KV cache grows, according to a dev.to walkthrough. Sizing a card by its weight file alone leaves that growing cache off the memory budget.
The Engineer · Build desk
What happened
- Most local-model sizing guides budget a single number, the model's weight size, and compare it against the GPU's VRAM.
- At fp16 a 7B model's weights take about 14 GB, and quantized to 4-bit they take 3.5 to 4 GB, depending on overhead.
- In the walkthrough, that 4 GB model loads with 12 or 16 GB to spare and still dies with an out-of-memory error once the conversation reaches the 30,000-token range.
- The KV cache is a second memory consumer that grows with every token the model reads or writes, while the weights stay fixed after loading.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure A load-and-ask acceptance test passes on hardware that will fail in real use, because the failure depends on how long the conversation runs.
- constraint Weight size alone cannot tell a buyer which GPU a long-context local deployment needs; the longest planned context has to be part of the sizing.
- decision Each local deployment has to choose between 4-bit weights with a quality penalty and higher-precision weights with a shorter usable context.
Weight memory is settled before the first prompt. "Loading a model onto a GPU is a one-time allocation: the file gets read, the parameters get placed in memory, and that number is locked in the moment the model finishes loading," the post's author wrote [1]. Guides budget the weights because they are "the only number that's fixed and printed on the model card before you ever run it," the author wrote [2].
The late timing comes from how memory gets allocated. According to the walkthrough, the crash happens on "a straightforward stack that allocates memory as the conversation grows," after twenty minutes and a few dozen turns with the same model, GPU and settings [3]. On that kind of stack the load succeeds because the cache is allocated later, turn by turn [3].
The post's own figures put a rough price on each token. With 12 to 16 GB free after a 4 GB load, a crash in the 30,000-token range implies the growing memory costs roughly 0.4 to 0.5 MB per token, assuming the cache took all of the headroom [1]. That figure belongs to this example. What carries over to other models is the division: free memory after load, divided by per-token cache cost, is the longest conversation the card can hold [1].
I'd give the cache its own line in the budget, next to weights and fixed overhead. Size it as per-token cost times the longest context the user will keep open. It is the only line that depends on how the model is used [5].
Quantization helps once. Taking the 7B weights from fp16 to 4-bit frees about 10 GB [2]. The author wrote that the cache "doesn't get quantized the same way" as the weights [6]. At the per-token cost above, those 10 GB buy roughly 19,000 to 25,000 more tokens of conversation, and then the same crash arrives [3].
The written post does not say what that room costs in quality. The video lists a 9:00 segment titled "The 4-Bit Quality Cliff" and a 4:32 segment on the cache math, but the text stops mid-sentence before either [8].
What to watch
- The per-token formula in the video's 4:32 segment, checked against the cost the 30,000-token crash implies.
- Whether the 9:00 'Quality Cliff' segment puts a measured number on 4-bit loss, and for which models.
- What the 11:15 segment says about why hosted APIs avoid this failure.