Build1 publisher3 min readPublished
vLLM measured its portability layer at 3.4 percent below native throughput on an H100
PyTorch says vLLM's frontier models now ship as hardware-specific flat definitions that torch.compile cannot trace, and the new HW agnostic layers are what users on other accelerators get instead. The overhead figure came from an H100.
The Engineer · Build desk
What happened
- PyTorch says vLLM is changing its internal implementation in ways that make it incompatible with fullgraph torch.compile, in pursuit of state-of-the-art performance on frontier models.
- Every frontier model added to vLLM in recent months uses the new hardware-specific flat definition, which replaces torch.compile with custom fusions and model-specific optimizations.
- A new set of HW agnostic layers is meant to preserve portability, and on NVIDIA H100 it comes within 3.4 percent of native total token throughput as a geometric mean across three recent models.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Portability stops being a property of the compiler and becomes a maintenance commitment: someone has to keep the HW agnostic layers in step with a frontier path that no longer has to compile.
- decision Maintainers of out-of-tree accelerator plugins now choose between tracking a refactor of the layers they override and pinning to a vLLM release that predates it.
- constraint Day-one support for a new frontier model is being written for specific hardware, so getting it onto a non-NVIDIA fleet is a second piece of work.
Fullgraph compilability demanded a specific discipline from anyone adding a model: compose it from the shared layers in `model_executor/layers`, and keep it traceable by Dynamo. Each new kernel had to be registered as a torch library op with a fake implementation and correct mutation annotations [8]. PyTorch's post calls that work "a tax on model development, and it is paid by whoever adds the model, including the advanced users who bring their own" [9]. Flat models drop all of it. They use custom fusions and other model-specific and hardware-specific optimizations, and they do not go through torch.compile [12].
Frontier open-weight architectures are diverging, shipping bespoke layers and their own optimized kernels, and the divergence now reaches attention itself: DeepSeek V4 and Kimi K3 get million-token context by completely different approaches [6][7]. Blackwell GPUs and rack-scale systems like the GB300 NVL72 also need careful kernel engineering to exploit new features and to overlap computation with communication [10]. The third reason in the post is coding agents. Claude Code and OpenAI Codex are effective at designing optimizations for a specific model on specific hardware. They work best when they do not have to worry about whether a change makes things worse for a different model on a different accelerator [11].
Figure 1 in the post shows all three flavours of model definition resolving to a single implementation of each common layer, with SpyreRowParallelLinear as an out-of-tree plugin overriding RowParallelLinear on one accelerator [16][15]. One resolution point made that override cheap: write a class, and every model that used RowParallelLinear picked up your kernel. Those same layers and ops are the ones likely to be refactored into a form incompatible with torch.compile [14].
Invert the H100 figure. A 3.4 percent throughput shortfall is about 3.5 percent more accelerator time for the same number of tokens (0.034 / 0.966). A load that fills 100 GPUs on the native path fills roughly 103.5 on the agnostic one [18][19]. For that to be the price in your deployment, the three models behind the geometric mean have to resemble what you serve. The overhead also has to behave on your accelerator the way it behaves on an H100 [4]. The post reports the number as a geometric mean across three recent models and does not name them [20].
I think the project made the right call. It says the effort is necessary to stay competitive on the latest GPU benchmarks [21]. The measurement comes from the hardware family least likely to need the portable path. The HW agnostic layers exist for people running AMD GPUs, Intel XPUs, Google TPUs, IBM Spyre, Huawei Ascend, older NVIDIA cards and more exotic models [5][2].
What to watch
- Whether vLLM publishes HW agnostic throughput numbers on a non-NVIDIA accelerator or an older GPU.
- Whether the refactor of model_executor/layers keeps a single override point for out-of-tree plugins.
- Whether the next frontier model of the DeepSeek V4 or Kimi K3 kind gets an HW agnostic definition as well as a flat one.