llama.cpp's -ngl flag keeps a 9B model on a 6GB card by leaving 28 layers on the CPU

The quant tag is the only part of a GGUF filename that changes what the GPU has to do. Run `ollama run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M` and Ollama resolves the Hugging Face repo, uses the tag after the colon to pick the file inside it, and serves the result ^[3]. It has been able to do that without you handling a file since 2024 ^[4]. Ollama does not let you choose how many layers go to the GPU ^[6].

That only matters when the file does not fit. The fitting rule in the guide is file size in GB plus about 1 to 2 GB of context overhead, inside VRAM ^[13]. It calls a 4.7 GB Q4_K_M of a 9B model comfortable on an 8 GB card ^[14]. Earlier it puts the same 9B Q4 at roughly 5.5 GB, down from about 18 GB at full precision ^[15]. Budget the larger file with the top of the overhead range and you sit at 7.5 GB of 8 ^[20]; the smaller one lands at 6.7 GB ^[21]. The 0.8 GB spread between the two published sizes is most of a gigabyte out of the one to two the rule tells you to hold back ^[22].

Q8_0 is where an 8 GB budget breaks. It is listed as near-lossless, and the guide says to use it "if your VRAM eats 8.5 bits per weight without noticing" ^[11]. At 8.5 bits, 9 billion weights come to about 9.6 GB before a single token of context ^[23]. The same sum on a 4B model gives about 4.25 GB, or 6.25 GB once you add 2 GB of context ^[24]. The advice to prefer a Q8 4B over a Q2 9B follows from those two figures ^[16]. The bottom of the ladder, Q2_K and below, is described as a model that starts degrading into nonsense mid-sentence ^[12], which shows up mid-generation and not at load time.

`llama-server -m ./gemma-2-9b-it-Q4_K_M.gguf -ngl 99` pushes 99 layers to the GPU, and setting the number lower leaves the remainder on the CPU ^[8]. The guide reports a 9B running on a 6 GB card at 20 of 48 layers offloaded, slower but working ^[9]. That is 28 layers doing CPU-speed work on every token ^[25]. A file you already downloaded loads in Ollama too, through a one-line Modelfile pointing at the path ^[7], with the layer count still out of reach ^[6].

The quality number needs more care than the size ones. Q4_K_M is described as within a percent or two of full-precision quality at roughly a quarter of the size ^[10], and the guide does not name a benchmark, an evaluation, or the models it was measured on ^[28]. For the figure to transfer, the measurement would have to be on work like yours: a percent or two of degradation spread across general text is not the same as a percent or two concentrated in the tokens that close a JSON object. The size half is checkable, and slightly generous. Roughly 5.5 GB against 18 GB is 31 percent, not 25 ^[26].

Conversion runs one way in practice. You cannot fine-tune a GGUF ^[18], and safetensors holds the full-precision weights that training and tools like transformers and ComfyUI expect ^[17]. Getting from those weights to something Ollama or llama.cpp will load takes llama.cpp's `convert_hf_to_gguf.py` ^[18].