Build1 publisher2 min readPublished
Pointing the Ray head and workers at vLLM's own image removes the numpy 2.0 ABI crash
A dev.to walkthrough of a 600GB NVFP4 model on discounted 8xH100 spot nodes traces the two crashes that arrive before the first prompt to a pip resolver replacing numpy and a KV cache sized for a million tokens.
The Engineer · Build desk
What happened
- A dev.to walkthrough documents deploying a roughly 600GB model, Inkling-NVFP4, on a discounted Spot A3 instance on Google Kubernetes Engine, and the deployment crashes before it serves a prompt.
- Installing vllm 0.25+ and transformers on the default rayproject/ray image upgrades numpy to 2.0, and the base system was compiled against numpy 1.0, so the nodes stop communicating and crash.
- Installing those packages at server start instead hits Ray's built-in 10-minute download timeout, which the post describes as tricky to debug and a noticeable hit to developer velocity.
- With the software fixed, the engine still crashes on boot: the default 1-million-word context asks about 7GB per GPU for KV cache on cards the post says are already 95% full.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The configuration that fits serves about 0.41% of the model's default window, so this discounted deployment is a 4,096-token service and long-context work needs different hardware or a smaller checkpoint.
- cost The out-of-memory crash lands after the weights are resident on the cards, so every debug iteration pays for a full 600GB load on rented A3 capacity before it tells you anything.
- precedent When the working answer is a vendor's serving image used as the cluster base image, the vLLM release cadence becomes the upgrade schedule for the platform team's Ray head and workers.
640GB of HBM across eight H100s is 80GB a card [2][1]. A 600GB model split eight ways is 75GB of weights on each [1][2]. The `gpu_memory_utilization=0.96` setting the post recommends caps vLLM at 76.8GB per card [9][3], leaving 1.8GB above the weights for KV cache, activations and engine workspace [4]. The default KV allocation asks for about 7GB per GPU [7]. That is close to four times what is free [5].
The 7GB is a claim about one model's attention layout. Multiply it out and 7GB on each of eight cards is 56GB for a million tokens, or roughly 56KB per token [6]. For that to be right the model has to store few KV heads per token, at low precision, or both. The post does not publish the layer count, KV head count or cache dtype, so the number should not be carried across to a different 600GB checkpoint.
KV cache scales linearly with sequence length, and so does the fix. Dropping the limit from 1,000,000 to 4,096 is a factor of 244, which takes 7GB down to about 29MB per GPU for one full-length sequence [8][7]. The post describes the result as "a few megabytes per GPU" [8]. Against 1.8GB of headroom, 29MB per sequence buys somewhere around sixty concurrent requests at the full 4,096 tokens [8].
The software failure is a resolver problem. The `rayproject/ray` base was compiled against numpy 1.0, and installing vllm 0.25+ and transformers on top of it pulls numpy 2.0 in underneath those builds [3]. The post explains this with two construction foremen who change dialect mid-shift; the pip dependency resolver is the less charming account. Doing the install when the server starts adds the second failure, because Ray's built-in download timeout is 10 minutes and the post says gigabytes of AI packages will almost certainly blow through it [4].
The write-up says that with `vllm/vllm-openai:v0.26.0` as the base image for both head and worker nodes, the servers boot "with no API conflicts, no ABI mismatches, and zero download timeouts" [5]. That image already contains ray, vllm, transformers and a numpy those builds agree on [5]. Run `pip install -U transformers` inside that pod later and you reopen the same resolver path that broke the Ray image [3].
What to watch
- Whether vllm/vllm-openai images after v0.26.0 keep shipping a Ray build the head and worker nodes can still cluster with.
- Whether the model's config is published, since the 7GB KV cache estimate depends on layer count, KV head count and cache dtype.
- What the deployment does when the Spot A3 node is preempted and 600GB of weights has to load onto the cards again.