The first hypothesis was the obvious one: a bf16 checkpoint meeting a chip with no bf16 datapath, so rewrite the checkpoint. It was converted host-side, the parameter dtypes came back as 541 float16 arrays plus one uint8 and one int8, and conversion still accounted for 54.0% of decode [9]. The 87% figure covers conversion plus an fp32 `gemvx` path [5] while the 54.0% is quoted for conversion alone, so subtracting one from the other only bounds what the rewrite bought: 33 points at most, with 54 still sitting in the compute path [2]. According to the dev.to write-up, the `gemvx` line is the diagnosis. XLA round-trips through fp32 whatever the file on disk says, and the round trip disappears only on a card whose compute dtype matches its storage dtype, which is what the g6 demonstrates [10]. That puts the fix in the process rather than the artifact: the port reads the live compute capability and sets `COMPUTE_DTYPE = float16 if IS_PRE_AMPERE else bfloat16` [11].
How much is recoverable is a separate question. The g5g runs at 26% of its own memory-bandwidth roofline and the g6 at roughly all of its own [6]. Take the reciprocal and the T4G carries about 3.85x of headroom against itself [1]. That is close to the measured 3.7x decode gap between the two chips [4], but the two numbers are different quantities. One compares a card to its own bandwidth ceiling; the other compares two cards. For a dtype-corrected T4G to land where the L4 sits, the two cards' absolute memory bandwidths would have to be comparable, and percentages of each card's own roofline cannot tell you that.
The 3.7x is a claim about one workload: `google/gemma-4-E2B-it` through a hand-written pure-JAX port with no PyTorch, no vLLM and no `torch_xla` [3], at 64 output tokens, concurrency 1, three repeats per cell, median reported [14]. Two conditions have to hold before it transfers. Decode has to be weight-bound rather than context-bound, which is what the sweep shows: decode moves 0.8% across a 50x context range on the g5g and 0.4% on the g6 [16], so the KV cache is not what sets decode speed on either box. And the engine has to be warm, because `max_new_tokens` is a `static_argnames` entry and `(bucket, max_tokens)` is the compiled shape; the first request on the g5g took 18.06 s against 4.50 s warm [18]. Read the end-to-end column instead of the decode gauge and the gap narrows with context, from 3.72x to 3.35x [3], which is prefill scaling linearly in the padded bucket rather than decode degrading [15].
Adoption cost on the older host is not the toolchain. `jax[cuda13]` ships CUDA as wheels including aarch64 builds, there is no build step, no CUDA toolkit and no Rust toolchain are required, and install took 117 s with the cache restore included [21]. The author reports the profile reproducing across a different instance, a different AMI and a restored cache at 1466.0 ms against 1467.1 ms [20], and flags that a 56x figure from an earlier first-serve baseline is TTFT specifically, not the whole-request ratio [19]. Both families ran on spot [1], so the dollar comparison stays an exercise for whoever has the billing console open.
