Build1 distinct publisher3 min readPublished
Each rig had been quoting throughput from a field its own server invents, so the vLLM comparison had never actually run. Switching to wire-level timing cost under three dollars of instance time and produced five corrections.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A gauge in a server's `usage` block is computed inside the process, from whatever boundaries its author picked. The portable alternative is the gap between tokens arriving on the wire, which every OpenAI-compatible server produces because they all stream [7]. The harness now takes that path using `vllm bench serve`'s exact TPOT definition, `(latency - ttft) / (output_len - 1)`, which is what makes its output comparable with that tool's published figures, and its `auto` mode probes an endpoint once to decide whether a server-side gauge exists at all [9][8].
Collecting both statistics at once put a size on the difference. One rig's gauge sat 2 percent off its stream number, another's 4.6 percent, on the same day and the same instance shape [10]. So the tempting repair, converting old `usage` figures with a ratio measured on a different rig, imports 2.6 percent of error into a table whose smallest interesting gap is 24 percent [11]. That is about a tenth of the distance you are trying to read [21]: survivable in a 24 percent gap, fatal in a 3 percent one.
The JAX rig failed differently. Its decode gauge was formatted to one decimal place, so at roughly 13 tok/s the tick size is 0.78 percent, and every triplicate cell it had ever produced read 12.8, 12.8, 12.8 [12]. Divide 0.1 by 13 and you get 0.77 percent, the same figure reached from the other direction [22]. Three repeats agreeing to the byte is the sort of result you put on a slide, right up until you do that division. The agreement is the mark of a quantiser coarser than the run-to-run noise, not evidence of a stable rig.
Memory is the one place where reasoning would have been enough. 9.5 GiB of float16 weights sit in 15,360 MiB of device memory, and KV costs about 18 KiB per token [16]. 15,360 MiB is 15 GiB, leaving 5.5 GiB, which is on the order of 320,000 tokens of cache before headroom becomes the problem, and that is before subtracting runtime overhead [23]. The cache was never going to be the binding constraint here.
What limits transfer is the model shape rather than the arithmetic. E2B-it is 2B effective out of about 5B total, and most of what stays resident is a per-layer-embedding table that decode reads as a gather, never streaming through a matmul [15]. A ranking measured on that access pattern says little about a matmul-bound serving mix, and nothing at all about batch behaviour at concurrency this experiment never reached.
One note on provenance. The two dev.to copies are the same write-up posted under different accounts, so the report has a single author despite appearing under two names. It counts five wrong claims; the excerpt documents three of them [4].
Ranked by verification strength, evidence, and original report placement.
The article is a step-by-step comparison of three Gemma 4 deployments on a single AWS-hosted GPU-enabled system, with a suite of Python MCP tools for managing each deployment and one benchmark harness shared across all three so that the runtime is the only variable.
Three rigs in one monorepo serve google/gemma-4-E2B-it on an AWS G5g instance: one runs vLLM, one a pure JAX port, one PyTorch with transformers. The hardware is identical and only the runtime slot moves.
The sweep script read its throughput figure straight out of the response body as usage.decode_tokens_per_second, a field the author's own servers invent. vLLM does not emit it, so the harness could not be pointed at the vLLM rig at all, and the three-way comparison had never actually been run.
The same write-up appears twice on dev.to, under the gde account and the aws-builders account, with identical text.
The whole exercise cost under three dollars: nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts.
The exercise also bought five wrong claims, each caught by measuring instead of reasoning; on hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
2 articles · September 1, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Fitting 6.155 GB of Gemma 4 weights lands you on AWS's only Arm GPU family1 distinct publisher
build
A Pallas kernel tiled for TPU VMEM splits one Gemma 4 port into two checkpoints1 distinct publisher
build
Emulating bfloat16 on a T4G burns 87% of decode on dtype conversion1 distinct publisher
build
Only the amd64 vLLM image carries the sm_75 kernels the cheapest AWS CUDA box needs1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Self-audited, checkable in principle
Every number traces to one author's own logs — and the second dev.to copy is the identical text under a different account, which buys reach rather than corroboration. What keeps this above anecdote is that the corrections are shown rather than asserted: the offending harness line, the one-decimal format string, the --decode-source help output, the verify_gpu_arch reading of 15,360 MiB. The stream statistic is also anchored to vllm bench serve's published TPOT definition, which someone else can hold this harness against. What is missing is the payoff table: the 24 percent gap that justifies the whole method change never appears as data in this text.
One monorepo, nineteen spot instances
Adoption stops at the author's own bench: three rigs in a single monorepo, nineteen g5g.2xlarge spot launches, about four and a half instance-hours, three sweeps and nine timed boots. No other team, product or user is reported running the harness or the MCP tooling, and the only third-party artifact touched is vllm bench serve's formula. The spot capacity exhaustion across all four zones is the one signal here about anyone else's demand for G5g.
Undersells its own transferable lesson
A headline about five wrong claims could easily be inflation; here the piece delivers a diagnosis for each and spends most of its length on the author's own mismeasurement, which is the opposite of a promotional posture. If anything it aims too low: framing this as one T4G's story buries the general finding that any two servers quoting home-grown throughput fields are not comparable, and that a gauge printed to one decimal can manufacture byte-identical repeats that read as reproducibility. The one place the framing outruns the shown evidence is the 24 percent gap, which arrives as a given.
Distribution stacking, self-critical content
The same post going out through both the gde and aws-builders accounts on dev.to is a reach play, and the piece is a showcase for the author's own MCP tooling and monorepo. Pulling the other way, hard: the substance is an inventory of the author's own broken measurements, including the admission that a headline three-way comparison had never actually been run. Nobody is paid in this story that the text discloses — no vendor, sponsor or pricing interest appears — so the pressure that matters is reputational, and it points toward candour rather than away from it.
Method solid, numbers unwitnessed
We can be fairly confident about what was done — the commands, the formula, the code changes are all on the page — and much less confident about the magnitudes, because a single author supplies them and no one has re-run anything. The arithmetic checks that can be done, do: 0.1 over 13 tok/s lands on the quoted 0.78 percent, 2.6 percent is about a tenth of 24 percent, under $3 across 4.5 instance-hours is a plausible spot rate. Confidence would move most if a reader posted a stream-mode sweep from their own G5g box.