Build1 publisher3 min readPublished
A tile-size clamp in vLLM's Triton kernel gets Gemma 4 running on a Tesla T4
Getting Gemma 4 E2B to serve on a 2019-era Tesla T4 under vLLM 0.29.0 turns on a clamp inside the Triton attention kernel, and the QAT checkpoint's advantage shows up in host memory before it shows up in tokens per second.
The Engineer · Build desk
What happened
- A dev.to guide deploys Gemma 4 E2B to a Tesla T4 under vLLM, managed by a suite of Python MCP tools rather than any provisioning step, since the GPU is already attached to the VM.
- After the CUDA 13 torch went in, verify_gpu_arch reported sm_75 in the published wheel's architecture list, along with capability (7, 5) and working fp16 matmul on the T4.
- The first server start was killed by the kernel out-of-memory handler while loading weights, naming process 14015 (VLLM::EngineCor) on a host with 7.8 GB of RAM and no swap.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The fix is an edit to an installed Python file, so every routine pip operation on vLLM can undo it. Any team adopting this pattern inherits a verification step it must run on every upgrade, not once at setup.
- cost Because sm_75 ships in the published wheel, the cost of putting a current small model on Turing silicon is a patched kernel file and a swapfile. No compile, and no custom wheel to maintain.
- decision On a two vCPU host, checkpoint choice is a host-RAM decision as much as a throughput one: the QAT weights staged with about 2 GB less swap pressure than bf16, which is what decides whether the engine survives loading.
vLLM selects its Triton unified attention backend because Gemma 4 mixes two attention widths, 256 in the sliding-window layers and 512 in the global ones [10]. At width 512 that kernel asks for more shared memory than Turing allows in a single block [10]. The rig's `apply_turing_patch` clamps the tile sizes in the installed vLLM so they fit, one occurrence in `/opt1/pyuser/lib/python3.13/site-packages/vllm/v1/attention/ops/triton_unified_attention.py` [10][11].
That edit sits in site-packages, so pip owns the file. Reinstalling vLLM removes the clamp [12]. The guide's answer is an interlock: `start_vllm_server` refuses to start until `verify_turing_patch` reports the clamp present [12]. It needs to be an interlock, because vLLM 0.29.0 pins torch to an exact version, `torch==2.13.0`, so upgrading vLLM moves torch underneath you as well [9].
The install traps come earlier and are cheaper. The box arrived with a CUDA build of vLLM 0.26.0 on torch 2.11.0+cpu, where `import vllm` fails on `libcudart.so.13` and the CPU torch reports an empty architecture list for every GPU [6]. A dependency resolver that considers both packages satisfied is not much help here. Installing the CUDA 13 torch with `--no-deps` skips cuDNN and the NVIDIA runtime wheels, and `import torch` then fails on `libcudnn.so.9` [7]. With the dependencies in place, `verify_gpu_arch` reports `sm_75` in the published wheel's architecture list, so Turing needs no source build [8].
Host memory is where the checkpoint choice first shows up. MemTotal on this n1-standard-2 is 7,614,824 kB, which is 7,436 MiB [4][1], and the VM ships with no swap [14]. The first start was killed during weight loading with anon-rss at 3,993,204 kB [13]. After a 16 GB swapfile went on the large volume [14], swap use peaked at 6,761 MiB while the bf16 server started and 4,770 MiB for QAT [15]. The bf16 peak is 91 percent of the machine's physical RAM [2]. QAT needed 1,991 MiB less, 29 percent lower [3].
The write-up's headline figure is a 1.79x decode speedup for QAT over bf16 on this T4 [16], which is the same token count in 56 percent of the wall time [4]. Several things have to hold for that to transfer. The kernel has to be the clamped one, since the measurement was taken on a Turing host that cannot run the unclamped tiles at all [10][12]. The stack has to be vLLM 0.29.0 with torch 2.13.0+cu130 [9]. The card has to be a T4 at the same 70 W power cap, 1590 MHz core and 5001 MHz memory [5]. And decode has to be measured apart from prefill, single stream, because concurrency changes what the decode phase is doing. The write-up does not include tokens per second, batch size or prompt length. The load-time saving is measured on this host; the decode ratio stays one host's number until the run detail is published.
What to watch
- Whether vLLM ships a Turing-safe tile size for mixed attention widths upstream, which would retire the local patch and its interlock.
- Whether the guide publishes tokens per second, batch size and sequence length behind the 1.79x decode figure.
- Whether the next vLLM point release re-pins torch again and sends the same verify chain back to step one.