Only the amd64 vLLM image carries the sm_75 kernels the cheapest AWS CUDA box needs
A dated pricing run puts the Graviton2-hosted G5g 20 percent below the Intel G4dn per hour, but that host resolves a container image with no kernels for its own T4G, so it compiles vLLM before serving a token.
The Engineer · Build desk
What happened
- A run dated 2026-08-30 priced every NVIDIA instance type in us-east-1 straight from the AWS Pricing API rather than documentation, returning 64 types before filtering down to whole-GPU instances.
- Among whole-GPU instances the Graviton2-hosted G5g came out cheapest, at 20 percent below the Intel-hosted G4dn per hour on demand and 59 percent below it on spot.
- Read from the registry config blobs, the arm64 vLLM image lists arch targets starting at 8.0 with sm_75 absent, while the amd64 image under the same tag on the same day carries it.
- On g5g.xlarge, with about 7.5 GiB usable host RAM, the kernel refused to map the 9.54 GiB Gemma 4 E2B checkpoint and vLLM crash looped with Cannot allocate memory (12).
- Three types priced below the G5g, at 0.2020, 0.2375 and 0.4750 per hour, are fractional L4 slices of 2,861 to 5,722 MiB, and none of them can hold the 10.2 GB checkpoint.
Why it matters
- What it costs cost A budget built from the hourly rate pays for a from-source vLLM build on first boot and a swapfile the pricier instance does not need, and the author puts the image packaging ahead of anything else as the cost driver.
- Who has to decide decision Choosing a host CPU architecture is now a choice about which container manifest resolves, which moves the decision to whoever owns the image rather than whoever owns the capacity plan.
- What it limits constraint Because the L4 slice ladder charges in proportion to framebuffer, there is no cheaper landing spot for a 9.54 GiB checkpoint further down the price list.
- What doesn't add up contradiction The mechanism is reproducible from digests and error strings, but the per-token ranking rests on tables the published text does not show, so one half of the argument can be checked today and the other cannot.
`docker pull vllm/vllm-openai` resolves the tag against the host architecture before anything CUDA-related happens, and the two platforms behind that one tag disagree about which GPUs they serve. The amd64 config blob (sha256:2286e8533ca8) lists `TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0` and contains sm_75 [9]. The arm64 blob (sha256:2a7cde230b59) starts at 8.0 and has no sm_75, same tag, same day [10]. So a Graviton2 host resolves, on architecture alone, to the image with no kernels for the accelerator attached to it, and the Dockerfile appends no `+PTX`, which removes the JIT fallback as well [11]. That box builds vLLM from source before it answers a request [11]. Billing runs during the compile.
The GPU itself checks out fine. vLLM independently sized the KV cache at exactly 329,579 tokens on both instances [8], which is about as close to proof of an identical GPU-side configuration as a running system will give you. The author's reading is that the mismatch is a packaging decision by the vLLM project rather than a property of either CPU, and the largest single cost difference between the families [12].
Invert the discounts and you get the number a capacity plan actually spends. Twenty percent cheaper per hour on demand [1] means the Intel box costs 25 percent more per hour [20]. Fifty-nine percent cheaper on spot [1] means the Intel box costs roughly 2.44 times as much, a 144 percent premium [21]. That premium buys an image that already has kernels in it.
The L4 ladder does not close the gap either. Take the largest of the three cheaper slices as the 0.4750 tier: g6f.4xlarge doubles both sides, 11,444 MiB against 5,722 and 0.95 against 0.4750 [23]. Framebuffer per dollar-hour is flat, and the 11,444 MiB slice leaves about 1,675 MiB above a 9.54 GiB checkpoint for KV cache and everything else [24]. The 10,246,621,918 bytes in the mmap error is the same 9.54 GiB object, not a second one [25].
On the host side the split is per-vCPU RAM: 4 GiB on G4dn against 2 on G5g [13]. That is why g5g.xlarge lands at about 7.5 GiB usable and errno 12 comes back from the mapping rather than from residency; a swapfile fixes it, and g4dn.xlarge's 16 GiB maps the checkpoint with no swapfile configured at all [14][15].
For the cost-per-token ranking to transfer to your workload, several things have to hold. You have to pull the published tag rather than build an arm64 image with 7.5 in the arch list or `+PTX` enabled, because that is the difference being priced [12]. Your model has to sit in the same 9 to 10 GiB band, since a smaller checkpoint moves the mmap ceiling out of the picture [14]. And your instances have to be short-lived enough that a from-source build is a real line item; on a box you keep for a month, the compile amortises and the 20 percent hourly discount is the number that wins [16]. The supplied text names the comparison but does not reproduce the throughput rows, so the mechanism is checkable and the ranking is not [22].
Worth noting what the deployment does well. The Hugging Face token is read from Secrets Manager at boot instead of user data, because instance metadata is readable by anything on the box, and the fetch is wrapped in `set +x` because the bootstrap runs under `set -x` and bash traces assignments with their values [18]. Access is SSM only, with no inbound SSH rule and no key pair [17]. The MCP layer is one Python file over stdio with three dependencies [19].
What to watch
- Whether vLLM adds 7.5 to the arm64 arch list or ships +PTX, which would erase the difference the article prices.
- A rerun of the same Pricing API pull on a later date, since the 59 percent spot gap is a single-day reading.
- Publication of the throughput rows behind the cost-per-token verdict, which the supplied text does not include.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence58
- Adoption27
- Hype gap+16
- Incentives44
- Confidence54
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Among instances that give a whole GPU rather than a fractional slice, the cheapest real CUDA GPU on AWS is the Arm box: 20 percent cheaper per hour on demand and 59 percent cheaper on spot.
- [2]
The same article text and headline appear on dev.to under two different accounts, xbill and aws-builders.
- [3]
The article deploys Gemma 4 E2B onto the two cheapest whole-GPU CUDA instances AWS sells and compares what they cost to run; everything was measured on 2026-08-30.
- [4]
G5g pairs a T4G with a Graviton2 host on aarch64; G4dn pairs a T4 with an Intel host on x86_64. Both carry the same generation of NVIDIA Turing silicon, the GPUs are effectively the same part, and the host CPU is the variable.
- [5]
Every NVIDIA instance type in us-east-1 was priced from the AWS Pricing API rather than from documentation, using get-products filters for Linux, shared tenancy, no pre-installed software and capacitystatus Used; 64 types came back.
- [6]
Three instances are cheaper still, at 0.2020, 0.2375 and 0.4750, but all three are fractional L4 slices with 2,861 to 5,722 MiB and none of them can map Gemma 4 E2B's 10.2 GB checkpoint.
- [7]
The cheapest fractional slice that could map the checkpoint, g6f.4xlarge at 11,444 MiB, costs 0.95 per hour.
- [8]
Read off the running instances, every measurable property matched, down to vLLM independently arriving at a KV cache of exactly 329,579 tokens on both.
- [9]
vllm/vllm-openai publishes one manifest list with two platforms that are not compiled for the same GPUs; the linux/amd64 config blob sha256:2286e8533ca8 has TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0 with sm_75 present.
- [10]
The linux/arm64 config blob sha256:2a7cde230b59 has TORCH_CUDA_ARCH_LIST=8.0 8.7 8.9 9.0 10.0 11.0 12.0 with sm_75 absent; same tag, same day, and only the amd64 image carries SM 7.5.
- [11]
The host architecture selects the manifest: an Intel host pulls kernels that run on its T4, while a Graviton2 host pulls an image with no kernels for its own GPU, and the Dockerfile sets no +PTX so there is no JIT fallback; that rig compiles vLLM from source before it serves a token.
- [12]
The author describes the missing sm_75 kernels as a packaging decision by the vLLM project, not a property of either CPU, and the single largest cost difference between the two instance families.
- [13]
G4dn gives 4 GiB of host RAM per vCPU; G5g gives 2.
- [14]
Gemma 4 E2B's checkpoint is 9.54 GiB; on g5g.xlarge, with about 7.5 GiB usable, the kernel declines to map it and vLLM crash loops with 'RuntimeError: unable to mmap 10246621918 bytes from model.safetensors: Cannot allocate memory (12)'.
- [15]
The failure is of the mapping rather than of residency, and a swapfile fixes it; g4dn.xlarge has 16 GiB and maps the checkpoint with no swapfile at all.
- [16]
The article states that which instance to rent depends entirely on whether you are buying hours or tokens, and that the Arm box's headline hourly price is the wrong number to buy on.
- [17]
Prerequisites include G family spot quota (16 vCPU for the walkthrough, 4 for g4dn.xlarge), a subnet, a security group, an instance profile with AmazonSSMManagedInstanceCore, and no inbound SSH rule and no key pair, with everything run over SSM.
- [18]
The Hugging Face token is stored in Secrets Manager under vllm/hf-token and fetched at boot rather than placed in user data, because instance metadata is readable by anything on the box; the fetch is wrapped in set +x because the bootstrap runs under set -x and bash traces assignments with their values.
- [19]
The MCP server is a single Python file started over stdio, with requirements limited to mcp, httpx and boto3.
- [20]
If the Arm instance is 20 percent cheaper per hour on demand, the Intel instance costs 25 percent more per hour than the Arm one.
- [21]
If the Arm instance is 59 percent cheaper on spot, the Intel instance costs about 2.44 times as much on spot, a premium of roughly 144 percent.
- [22]
The supplied text refers to its comparison tables but their rows do not appear, so no tokens-per-second or cost-per-token figures are given in the material available.
- [23]
Taking the largest of the three cheaper slices as the 0.4750 per hour tier, g6f.4xlarge doubles both sides of the ratio: 11,444 MiB is exactly twice 5,722 MiB and 0.95 is exactly twice 0.4750, so framebuffer per dollar-hour is flat between those two tiers.
- [24]
The 11,444 MiB g6f.4xlarge slice leaves about 1,675 MiB above the 9.54 GiB checkpoint for KV cache and activations.
- [25]
The 10,246,621,918 bytes named in the mmap error equals 9.54 GiB, so the error refers to the same checkpoint file the article sizes at 9.54 GiB.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toThe Cheapest CUDA GPU on AWS Has an Arm CPU — and You Probably Want the Intel One
2 articles · August 30, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.