The fused W4A16 path is written in Pallas and tiled for TPU VMEM, which gives 16 MB per core, and at this model's shapes the tiles want 550 KiB to 1.1 MiB per block [16]. Lower that to a GPU and Pallas goes through Triton, where per-block scratch is shared memory, 64 KiB of it on Turing [17]. Against that ceiling the tile range is 8.6x to 17.6x over budget [23]. The two budgets are not even the same unit of accounting, since VMEM is counted per core and shared memory per block, and the nominal gap between the two figures is 256x [24]. Ada lifts the ceiling and still does not come near a megabyte [17]. `check_w4a16_fits_scoped_memory()` runs that comparison at startup and refuses with the arithmetic attached instead of dying at the first token [18]. The author's one-line version is that Pallas is portable as an API and not portable as a memory model [20]. Measured against the project's own bar, that the only thing changing between rigs is a config file, that is one leak [3].
The second leak does not announce itself. A wrong compute dtype does not raise; it emulates, and XLA routes bfloat16 on a pre-Ampere GPU through fp32 with nothing red in the logs [11]. On the SM 7.5 card the policy resolves to float16, Turing's only real 16-bit datapath, since the part has neither bf16 nor fp8 [13]. The first line the process emits carries the whole decision, including `pallas_interpret=False` [14], which the write-up notes is the difference between serving and silently running a simulator [15]. A simulator that answers your HTTP requests is the most courteous way to be wrong.
What kills the vLLM path on Turing is a shared-memory request of 98,304 bytes against a 65,536-byte hardware limit [9], which is 1.5x the budget [22]. That figure travels only if your rig makes the same backend selection at the same shapes on a card with 64 KiB of shared memory per block. Somewhere with a higher ceiling the same code may fit and tell you nothing about the T4G. In this port there is no vLLM in the path at all, and attention is ordinary XLA rather than a hand-tiled kernel, so the attention path has no per-block ceiling to exceed [1][10].
The geometry is still work for whoever ports next. The 8:1 MQA means the KV budget bears no relation to what the parameter count implies [5], and a KV-share map folds 35 layers onto 15 caches, an average of 2.3 layers per cache [6][25]. On top of that sit a 512-slot sliding ring and a 4.70 GB per-layer-embedding table that is quantized to 4 bits on load [7]. The write-up reports a week lost to a padding-eviction bug in that ring, under an invariant that a cache index is an absolute real position and that padding never occupies an index a real position uses [21]. The text breaks off mid-sentence there [26]. This is one engineer's set of rented rigs, published on dev.to [27], so read the memory ceilings as checkable arithmetic and the lost decode throughput as an unquantified characterisation.
