One environment variable swaps Gemma 4 to its 2.05x-faster QAT weights on a SageMaker L4

The 2.05x figure comes from a decode measurement designed to survive a noisy client ^[3]. Each endpoint returns fixed-length replies of 16 and 512 tokens, five of each, at temperature 0 with ignore_eos set ^[3]. The rate is (512 - 16) divided by the difference between the two median times ^[3]. Subtracting the 16-token time cancels the aws CLI start-up and the network round trip ^[3]. It is a careful design. If each CLI call had been timed end to end, both checkpoints would carry the same fixed overhead and the ratio would be pulled toward 1.

It is also a single-stream number. The decode test is separate from the parallel one, so it describes one caller waiting on one long answer ^[3][4]. The same script runs 1, 4 and 16 simultaneous requests of 256 tokens, plus a 40-question exact-answer quiz ^[4][5]. The available text of the post ends at the command that runs the suite, before either result. For the 2.05x to hold on another endpoint, that endpoint's traffic has to resemble the test: E2B, one L4, long replies, few requests in flight ^[1][3].

The quiz is small. It holds 15 two-digit multiplications, 15 three-number sums and 10 capitals, scored by regular expression ^[5]. Each question moves the score by 2.5 points ^[6]. A checkpoint that broke arithmetic would fail it, but the quiz does not test instruction following at all.

Memory is the result in hand. Loading the bf16 weights took 9.75 GiB and 82.75 seconds, and the QAT weights took 8.01 GiB and 66.19 seconds ^[7]. The 1.74 GiB difference ^[11] matches the 18% saving the author reports ^[8]. That saving is modest for 4-bit weights because only the transformer body is quantized, according to the checkpoint header ^[8]. vLLM gives the freed memory to the KV cache. The cache grows from 723,484 to 867,999 tokens, about 20% ^[9][10]. At 8,192 tokens per request, vLLM reports room for 105.96 concurrent requests against 88.32 ^[9].

Getting the QAT build running takes few steps, but only one route works. JumpStart lists seven Gemma 4 packages and none is a Google QAT checkpoint ^[13], so the route is AWS's vLLM container with a Hugging Face model ID ^[1]. Google trains the checkpoint with 4-bit weights in the loop ^[20]. It publishes four QAT forms of the model, and only google/gemma-4-E2B-it-qat-w4a16-ct loads in vLLM ^[14]. The suffix means 4-bit weights and 16-bit activations, exported in compressed-tensors format ^[14]. The swap is one environment variable, SM_VLLM_MODEL, and vLLM confirms it at start-up with one log line: quantization=compressed-tensors ^[15].

Capacity limited the experiment more than the model did. The post's prerequisite is a quota of at least one ml.g6.xlarge in a region with L4 capacity ^[19]. The account holds one instance for each single-L4 type in us-east-1, us-east-2 and us-west-2 ^[16]. Requests in us-east-1 and us-west-2 sat in Creating for about 30 minutes, then returned InsufficientInstanceCapacity ^[17]. Half an hour is a long wait for a no. In us-east-2, an L4 was placed within minutes every time ^[17]. An InstancePools fallback list holds quota for every type it names, so two endpoints cannot share one region's single L4, and the checkpoints ran one after the other ^[18].

An account in that position can only serve more traffic by getting more out of each instance. I'd put E2B on an L4 on the QAT checkpoint today. I would not call it the default until its quiz score matches bf16's ^[5].