Emulating bfloat16 on a T4G burns 87% of decode on dtype conversion
Identical code and identical weights diverge 3.7x on decode between a T4G and an L4, because Turing has no bf16 datapath and XLA emulates one in fp32 while the instance keeps returning 200s and green health checks.
The Engineer · Build desk
What happened
- A dev.to write-up served a byte-identical payload, same build id 51bc52c9e2e9 and the same 6,155,450,950 weight bytes, on two AWS GPU instances, changing only the chip and its host.
- Steady-state decode on the g6 with an Ada L4 came in 3.7x faster than the g5g with a Turing T4G, running the same code against the same weights.
- An xprof profile over 20 decode steps with the service stopped attributed 87% of g5g decode to dtype conversion plus an fp32 gemvx path rather than to math.
- Turing has no bf16 or fp8 datapath, and a bfloat16 compute dtype does not raise on it: XLA routes the work through fp32 and emulates the type.
- Through every slow run the g5g returned HTTP 200, valid completions and a healthy /health endpoint.
Why it matters
- What it costs cost Equalling one g6's decode on the older family takes roughly 3.7 instances, so any hourly saving on g5g has to beat a 3.7x instance multiple before it saves anything.
- Who has to decide decision A fleet that mixes pre-Ampere and Ada cards has to choose compute dtype per device at startup instead of pinning it in one shared config, because the wrong choice returns correct answers.
- What it limits constraint Substitution is bounded by request shape as well as speed: prompts above the g5g's 4,096-token usable context fail on a prefill transient, so the older family cannot absorb the same traffic mix.
- What's now possible capability Printing compute_capability and compute_dtype on the server's first line turns this from a throughput mystery into a grep, which is the only supervision available for a fault that answers with 200.
The first hypothesis was the obvious one: a bf16 checkpoint meeting a chip with no bf16 datapath, so rewrite the checkpoint. It was converted host-side, the parameter dtypes came back as 541 float16 arrays plus one uint8 and one int8, and conversion still accounted for 54.0% of decode [9]. The 87% figure covers conversion plus an fp32 `gemvx` path [5] while the 54.0% is quoted for conversion alone, so subtracting one from the other only bounds what the rewrite bought: 33 points at most, with 54 still sitting in the compute path [24]. According to the dev.to write-up, the `gemvx` line is the diagnosis. XLA round-trips through fp32 whatever the file on disk says, and the round trip disappears only on a card whose compute dtype matches its storage dtype, which is what the g6 demonstrates [10]. That puts the fix in the process rather than the artifact: the port reads the live compute capability and sets `COMPUTE_DTYPE = float16 if IS_PRE_AMPERE else bfloat16` [11].
How much is recoverable is a separate question. The g5g runs at 26% of its own memory-bandwidth roofline and the g6 at roughly all of its own [6]. Take the reciprocal and the T4G carries about 3.85x of headroom against itself [22]. That is close to the measured 3.7x decode gap between the two chips [4], but the two numbers are different quantities. One compares a card to its own bandwidth ceiling; the other compares two cards. For a dtype-corrected T4G to land where the L4 sits, the two cards' absolute memory bandwidths would have to be comparable, and percentages of each card's own roofline cannot tell you that.
The 3.7x is a claim about one workload: `google/gemma-4-E2B-it` through a hand-written pure-JAX port with no PyTorch, no vLLM and no `torch_xla` [3], at 64 output tokens, concurrency 1, three repeats per cell, median reported [14]. Two conditions have to hold before it transfers. Decode has to be weight-bound rather than context-bound, which is what the sweep shows: decode moves 0.8% across a 50x context range on the g5g and 0.4% on the g6 [16], so the KV cache is not what sets decode speed on either box. And the engine has to be warm, because `max_new_tokens` is a `static_argnames` entry and `(bucket, max_tokens)` is the compiled shape; the first request on the g5g took 18.06 s against 4.50 s warm [18]. Read the end-to-end column instead of the decode gauge and the gap narrows with context, from 3.72x to 3.35x [23], which is prefill scaling linearly in the padded bucket rather than decode degrading [15].
Adoption cost on the older host is not the toolchain. `jax[cuda13]` ships CUDA as wheels including aarch64 builds, there is no build step, no CUDA toolkit and no Rust toolchain are required, and install took 117 s with the cache restore included [21]. The author reports the profile reproducing across a different instance, a different AMI and a restored cache at 1466.0 ms against 1467.1 ms [20], and flags that a 56x figure from an earlier first-serve baseline is TTFT specifically, not the whole-request ratio [19]. Both families ran on spot [1], so the dollar comparison stays an exercise for whoever has the billing console open.
What to watch
- Whether a post-fix decode number on the g5g with float16 compute recovers the headroom the 26% roofline implies; the write-up publishes the diagnosis, not the corrected throughput.
- Whether vLLM and PyTorch serving paths select float16 themselves on pre-Ampere cards, or inherit the same fp32 round trip from a shared bf16 config.
- Spot prices for g5g.2xlarge and g6.2xlarge, which decide throughput per dollar; the comparison here is per instance.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence62
- Adoption12
- Hype gap−5
- Incentives30
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
g5g.2xlarge pairs a Graviton2 (aarch64) host with an NVIDIA T4G, Turing, SM 7.5; g6.2xlarge is x86_64 with an NVIDIA L4, Ada, SM 8.9. Both were run on spot.
- [2]
The serving payload was byte-identical on both instances: build id 51bc52c9e2e9, config ple4 + int8_lm_head, and tpu_jax_weight_bytes reading 6,155,450,950 on each. Only the chip and its host differ.
- [3]
The workload is google/gemma-4-E2B-it, the dense reference checkpoint, served through a hand-written pure-JAX port with no PyTorch, no vLLM and no torch_xla.
- [4]
Decode throughput differs by 3.7x between the two instances, for the same code and the same weights.
- [5]
Profiling with xprof over 20 decode steps with the service stopped shows 87% of decode on the g5g is not math: it is dtype conversion plus an fp32 gemvx path.
- [6]
The g5g runs at 26% of its own memory-bandwidth roofline; the g6 runs at roughly all of it.
- [7]
Turing has neither bf16 nor fp8; its only real 16-bit datapath is float16.
- [8]
A wrong compute dtype does not raise, it emulates: bfloat16 on a pre-Ampere GPU does not fail, XLA routes it through fp32 and decode disappears into conversion.
- [9]
The checkpoint was converted to float16 host-side and re-run; parameter dtypes read {'float16': 541, 'uint8': 1, 'int8': 1} and conversion stayed at 54.0%, so storage dtype was never the problem.
- [10]
The fp32 gemvx line is the tell: XLA was round-tripping through fp32 regardless of what the file on disk said, and only a card whose compute dtype matches its storage dtype removes it, which is what the g6 shows.
- [11]
The port reads the live compute capability off the device rather than trusting a config file: COMPUTE_DTYPE = float16 if IS_PRE_AMPERE else bfloat16.
- [12]
The server states its device policy on the first line it emits, for example: platform=gpu compute_capability=8.9 compute_dtype=bfloat16 pallas_interpret=False, so a misconfigured instance is one grep away.
- [13]
The g5g serves correctly the entire time: it returns HTTP 200, valid completions and a healthy /health, and nothing in any log, metric or health check reports the conversion loss.
- [14]
Benchmark method: 64 output tokens, concurrency 1, 3 repeats per cell, median reported; the gauge is the engine's steady-state decode counter and end-to-end is wall time over the whole request including prefill.
- [15]
End-to-end figures fall from 12.43 to 8.22 on the g5g and from 46.23 to 27.55 on the g6 across the context sweep; that fall is prefill being linear in the padded bucket, not decode degrading.
- [16]
Decode moves 0.8% across a 50x context range on the g5g and 0.4% on the g6, a shape produced by a cost proportional to the weights rather than the context, which is why the KV cache is not what sets decode speed on either instance.
- [17]
Usable context on the g5g is MAX_MODEL_LEN=4096: 4,105 prompt tokens serve and 5,120 fails on a prefill transient.
- [18]
max_new_tokens is a static_argnames entry, so (bucket, max_tokens) is the compiled shape and the first request off a fresh engine pays XLA compilation; on the g5g that first request took 18.06 s against 4.50 s warm, a 4.0x whole-request ratio, so a harness that skips warm-up misreports the instance by a factor of four.
- [19]
The 56x figure quoted from the earlier first-serve baseline is TTFT specifically, which is a different measurement and not interchangeable with the whole-request ratio.
- [20]
The measurement reproduces: the same profile on a different instance, a different AMI and a restored cache landed at 1466.0 ms against 1467.1 ms.
- [21]
There is no build step on either instance; jax[cuda13] ships wheels carrying CUDA including aarch64 wheels for the Graviton2 host, no CUDA toolkit and no Rust toolchain are needed, and install took 117 s with the cache restore included. The aarch64 and x86_64 Deep Learning base AMIs are not interchangeable.
- [22]
Running at 26% of its own memory-bandwidth roofline implies the g5g has roughly 3.85x of headroom against itself.
- [23]
On the reported end-to-end figures the g6 lead narrows from 3.72x at the short context to 3.35x at the long one.
- [24]
Rewriting the checkpoint to float16 accounts for at most 33 percentage points of the reported overhead, leaving 54 points in the compute path; the two figures are not strictly comparable because 87% covers conversion plus the fp32 gemvx path while 54.0% is quoted for conversion alone.
- [25]
Matching one g6 instance's decode throughput on the older family takes roughly 3.7 g5g instances.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.tog5g vs g6 for LLM Serving: the Same Code, and 3.7x the Throughput
1 article · August 30, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.