Repacked QAT weights give Gemma 4 26B 15.6 times the FP8 build's KV cache on one TPU v6e

Google's "unquantized" QAT export of the 26B stores bf16 tensors, but the values came out of training for Q4_0 ^[8]. According to the author, every group of 32 weights along the input dimension already sits on a 16-level grid, a step times an integer level from -8 to 7 ^[8]. The group size was measured. Sampled groups of 64 fail the same test ^[8]. "That makes a 4-bit checkpoint a change of container," the author wrote ^[18].

The step is where a naive repack breaks. The textbook Q4_0 rule sets the step to max|w| / 8, on the assumption that the group's largest weight sits at level 8 ^[9]. When a group's peak sits at level 5, the rule derives 5/8 of the true step and re-rounds every weight onto a grid that does not contain it ^[9]. The author puts the damage at about 5% median error per group, and every shape check still passes ^[9].

The repack instead tries max|w| / m for m from 1 to 8, keeps the first step that reproduces the whole group, and refines it by least squares over the 32 values ^[10]. A tensor with any group off the grid would have stayed bf16, and none did ^[12]. The run wrote 15.29 GiB in 31 shards in 388 seconds on a 16-core machine with 15 GB of RAM, streaming one layer at a time ^[11]. The bf16 export is 48.07 GiB, 3.1 times larger ^[5][6]. A separate verify pass rereads both checkpoints from disk and checks, for every group, that the stored level is the source value's rank on the stored grid ^[15]. I like this check. It tests the exact property the conversion depends on, against the bytes that landed on disk.

The experts ship fused, with experts.gate_up_proj shaped [128, 1408, 2816]. The repack writes one module per expert, the layout vLLM already reads for int4 mixture-of-experts checkpoints ^[14]. Attention, the dense MLP and all 3,840 experts are quantized; the router, embeddings, norms and vision tower are copied unchanged ^[13]. On an NVIDIA L4 the checkpoint loads unpatched on vLLM 0.30.0 ^[4]. On TPU it needs one new method in vLLM's TPU backend, carried on the author's gemma4-w4a16-moe branch for #3660 ^[4][16].

The serving numbers follow from memory. One v6e has 28.74 GiB of usable HBM ^[5]. RedHat's FP8 build fits with 0.75 GiB to spare ^[7]. The QAT build uses 17.43 GiB, per the author, leaving 11.31 GiB ^[1][3]. That spare memory is 15.1 times the FP8 build's, close to the 15.6 times the reported cache sizes give ^[3][1]. At 2,048 tokens per request, 53,888 tokens holds about 26 requests. The FP8 build holds one and a half ^[4][7]. Output throughput is 1,283 tokens per second against 668, a factor of 1.92 ^[1][2].

I'd expect most of that 1.92 comes from batch depth, since the FP8 build cannot keep two 2,048-token requests resident ^[7]. If so, the gain transfers to traffic whose concurrency was capped by cache, and it shrinks for single-stream use. The accuracy evidence is one 3,880-record classification suite, where the two builds land within a point ^[3]. A classification label is a short output. Before trusting the build for long generation I would want a generation eval. The author committed every per-record output, log and script, so both comparisons can be rerun ^[17].

The writeup does not include a chip price. On the same chip at the same hourly rate, cost per output token falls to about 52% of the FP8 build's, if the throughput gain holds on a given workload ^[5]. Google's model card lists a w4a16 compressed-tensors build "for native, optimized inference with vLLM" for E2B, E4B, 12B and 31B ^[6]. For the 26B, Google did not publish one ^[6]. The int4 26B for vLLM is a community checkpoint, published as xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct ^[16].