Build1 publisher2 min readPublished
vLLM 0.30.0's weight cache serves a cached model to any checkpoint with the same layout
vLLM 0.30.0's weight cache fingerprints only safetensors headers, letting an engine on an altered Qwen3-0.6B run the daemon's original weights. Fine-tunes share their base model's headers, so a mix-up there would produce fluent wrong answers under a log line reporting success.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- The tester zeroed the 2,048 bytes of model.norm.weight in a copy of Qwen3-0.6B, leaving the safetensors header untouched.
- Loaded from disk, that copy answered the prompt "The capital of France is" with twelve exclamation marks.
- Through a daemon holding the original, the copy answered "Paris" and logged that it mapped 339 tensors, with no warning from either process.
- An engine pointed at a small random Qwen3 with different tensor shapes was rejected on the checkpoint field and fell back to loading from disk.
- The reproduction follows an upstream report of the same behaviour, filed as vllm-project/vllm#59647.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure A host serving a fine-tune through a daemon loaded with its base, or with an older step, would return plausible answers from the wrong model while the engine's target and its logs name the right one.
- constraint Comparing two training steps, or a base against its fine-tune, through one cache daemon is invalid on 0.30.0, since both sides present the same key and get whichever weights the daemon holds.
- decision On 0.30.0, choosing ipc_cache for fast restarts on same-layout checkpoints trades restart time against certainty about which weights produced an answer.
The key is built in vllm/model_executor/model_loader/weight_cache/protocol.py. Its hash_checkpoint function sorts the shard files by base name, then feeds each file name and its safetensors header into one hasher [3]. A safetensors header is a JSON index of each tensor's name, dtype, shape and byte offsets [3]. The tensor values never reach the hasher [3]. The other fields of WeightCacheKey are architecture, dtype, quantization, tensor-parallel rank and vLLM version. According to the write-up, a local base model and its fine-tune agree on all of them too [11].
The shortcut has a stated purpose. The docstring says the function "Hashes each shard's safetensors header so a daemon and an engine pointing at identical weights in different directories produce the same key" [10]. Keeping the directory out of the key is good design, because a copied checkpoint should hit the cache. Substituting the header for the bytes is the flaw. I'd expect the authors wanted to avoid reading every weight at engine start, since skipping that read is the reason the daemon exists [1].
The reproduction, published on dev.to, changed one 2,048-byte norm vector [6]. That is a cruder change than a fine-tune, and the fine-tune case was not run. It rests on the write-up's point that fine-tuning changes none of the header fields [12]. Shard file names also go into the hash. A collision therefore needs matching names as well as matching headers, and a fine-tune saved with its base model's sharding has both [1]. The test ran on an 8 GB RTX 2070 in fp16 [5], but the fault is in a hash function and does not depend on the card.
The feature page on main promises a "Safe fallback" for cached weights that "don't match the engine's configuration" [13]. The same page says the fingerprint covers "checkpoint content (hashed from safetensors metadata, ...)" [13]. The parenthesis is the most accurate line on the page. That page is not in the 0.30.0 tag [13]. The `vllm preload` command in the upstream report is not in the 0.30.0 CLI either, so the reproduction launched the daemon as a Python module [14].
The proposed fix, #59648, is titled "Fingerprint sampled tensor bytes, not headers only" [16]. Sampling reads some values at engine start and leaves the rest unread. A fine-tune that moves only a few tensors would be caught only if the sample covers them, and the title does not say how the sample is drawn. In the same test, load_format=auto loaded each checkpoint as itself [7].
What to watch
- Whether #59648 is merged, and how many bytes per tensor its sampled fingerprint reads at engine start.
- Whether the preload docs page reaches a tagged release with the "Safe fallback" wording narrowed to layout changes.
- A reproduction using a real base model and its fine-tune with matching shard names, rather than a zeroed norm vector.