Build1 publisher2 min readPublished
A 70B model gets 2.74 bits per parameter on a 24 GB card before anything else loads
The weights formula is the easy part of sizing local inference. Quantization metadata, the KV cache and the runtime's own buffers decide whether a 70B model fits, and a dev.to walkthrough shows where the advertised bit width stops helping.
The Engineer · Build desk
What happened
- A dev.to walkthrough on sizing GPUs for a 70B model names four inputs to any VRAM estimate: the model weights, the quantization format, the KV cache and runtime overhead.
- Its weights formula puts a 70B model at about 140 GB in FP16, which means even an 80 GB accelerator cannot hold the raw weights in memory.
- At roughly 4 bits per parameter the same weights come to about 35 GB, which the post calls practical on configurations with around 48 GB of GPU memory.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A 70B build that stays entirely on a 24 GB card has to average under 2.74 bits per parameter, and the cache and buffers come out of the same 24 GB, so the usable ceiling is lower than that.
- decision A 32 GB buyer has to pick a specific aggressive quantization of the model they want and accept whatever fidelity it costs, because the ordinary 4-bit build will not stay resident.
- cost Offloading layers to system RAM is paid for on every token the model emits, not once at load time, because decode revisits the weights each step.
- exposure A 22 GB model on a 24 GB card loads fine and then fails later in a long conversation, once the cache and the runtime's own allocations grow past the 2 GB of slack.
Invert the weights formula and the sizing question gets easier to argue about. A 24 GB card is 24 billion bytes, or 192 billion bits. Spread that across 70 billion parameters and you get 2.74 bits each [16]. That figure assumes the card holds nothing but weights, and the runtime also loads the KV cache, CUDA buffers, temporary tensors and kernels [14].
A 32 GB card allows 3.66 bits per parameter under the same assumption [17]. The dev.to post says a straightforward 4-bit 70B will generally still exceed 32 GB before any of the additional memory requirements are counted, and that more aggressive quantizations can bring some 70B-class models within range [12]. So a 32 GB buyer needs a specific sub-3.5-bit build of the model they want, and needs that build to hold quality.
The advertised bit width is a floor. A model described as 4-bit does not necessarily occupy 35 GB, because the format also carries scales, metadata and higher-precision tensors, and two formats both called 4-bit can differ noticeably in what they resident-load [6]. The post quantifies neither that overhead nor the bytes per token of KV cache, so the labels support a lower bound and nothing tighter.
This is where the 22 GB download on a 24 GB GPU goes wrong [14]. It leaves 2 GB for five other kinds of allocation [19]. The cache grows with the conversation, and the post contrasts a model that fits comfortably at 4,096 tokens with the same model at 32,768 and at 131,072 [13].
Offload is a capacity fix. A 24 GB card paired with 64 GB of system RAM potentially holds a quantized 70B across the two pools, with some layers on the GPU and the rest in normal system memory [9]. Decoding touches weights repeatedly as each token is generated, so any layer living in system RAM crosses back to the GPU on every token, over a bus with dramatically lower bandwidth than VRAM, and generation can slow considerably [10]. The post wrote: "But capacity and performance are two different problems." [11]
On an 80 GB accelerator, INT8 weights at roughly 70 GB leave about 10 GB for the cache and the runtime [18]. FP16 is not on the table at this parameter count, since 140 GB of weights does not fit in 80 GB [3]. The post's rule for choosing among the rest: "Use the highest-quality quantization that fits comfortably within the hardware you have." [15]
What to watch
- Whether splitting a 4-bit 70B across two 24 GB cards beats a single 48 GB card once inter-GPU traffic is measured.
- Offload benchmarks that report tokens per second alongside the exact layer split, not just that the model ran.
- Whether 70B-class builds below 3.5 bits per parameter appear and hold quality well enough for 32 GB cards.