Build Not yet confirmed elsewhere 1 publisher 2 min read Published Aug 28, 2026, 7:41 PM PDT

Fitting 6.155 GB of Gemma 4 weights lands you on AWS's only Arm GPU family

The cheap fractional-GPU tiers top out at 3 GB of device memory, so the weights pick the instance and hand you a Graviton2 host with a 2018 Turing card. The interesting cost is the toolchain, not the hourly rate.

The Engineer · Build desk

Send a correction

Illustration accompanying Fitting 6.155 GB of Gemma 4 weights lands you on AWS's only Arm GPU family — Generated illustration

What happened

A dev.to walkthrough serves Google's instruction-tuned google/gemma-4-E2B-it on an AWS G5g instance, which pairs a Graviton2 Arm host with an NVIDIA T4G and is the only Arm-based GPU family AWS rents.
Priced in us-east-1 on Linux on-demand against the Pricing API on 2026-08-28, g5g.xlarge at $0.4200 is the floor for a whole NVIDIA GPU, with g4dn.xlarge next at $0.5260.
The two cheaper GPU instances cannot hold the model: g6f.large at $0.2020 is an eighth of a GPU with 3 GB against 6.155 GB of weights, and g4ad.xlarge at $0.3785 carries an AMD card.
The rig resolves its image at launch from the arm64 OSS NVIDIA driver SSM parameter, because AWS also ships ARM64 DLAMIs built for Graviton CPU inference that boot perfectly and have no GPU.

Why it matters

What it limits constraint A 3 GB slice cannot hold a 6.155 GB checkpoint at any hourly rate, so device memory selects the instance type and price only breaks ties among the boxes that fit.
What it costs cost Buying a fractional-GPU instance large enough to hold the weights costs about 1.7x this rig per hour, which means the discount tier is cheap mainly for models you are not serving.
What's now possible capability Because the control plane's 105 tests run fully offline with every cloud and network boundary mocked, the operator can validate the whole launch path before a single GPU-hour is billed.
Who's exposed exposure With no inbound SSH rule and no private key, the entire administrative surface collapses onto IAM and SSM Run Command, and anything running on the instance can read its metadata.

Next story · Product · Oct 4

Trump's AI accord puts three of its four layers of control inside the companies it covers

Confirmed16 publishers

The pip story is the good news on this axis. On NVIDIA hardware JAX's wheels bring the CUDA libraries with them, so there is nothing to build ^[13]. Everything above that layer is where the aarch64 tax shows up. The engine in this rig is the repo's own Gemma 4 port, driven by a JAX generation loop behind an OpenAI-compatible FastAPI server under systemd ^[14]. Writing your own decode loop is a real line item, and it is roughly what you pay for a pairing the guide describes as one almost nothing in the ML ecosystem targets ^[4].

The Hugging Face token, incidentally, is written to Secrets Manager under the path `vllm/hf-token` ^[17], which on a pure-JAX rig reads like a fossil of the attempt before this one.

Then the memory budget. The T4G reports 15,360 MiB of device memory rather than the nominal 16 GB, and the guide says to budget against the measured number ^[11]. The weights are 6.155 GB ^[6]. Read as decimal gigabytes, that is about 5,869 MiB resident, leaving roughly 9,491 MiB for KV cache, activations and XLA scratch ^[24]. Read it as GiB instead and about 430 MiB of that goes away ^[24]. Workable, and not roomy, and the T4G has no fp8 ^[3], so the usual lever for buying cache space back with a smaller numeric format is absent.

Now read the default. The rig runs `g5g.2xlarge` at 1 GPU, 8 vCPU and 16 GiB, and that instance type is hardcoded into the MCP server's environment block ^[10]^[16]. The guide's own note that the first fitting fractional instance, `g6f.4xlarge` at $0.9500, is 1.7x this rig ^[7] puts the configuration you actually launch at about $0.56 an hour ^[23], which is around 6% above the $0.5260 `g4dn.xlarge` that the family undercuts on paper ^[25]. It is a bigger box than the `g5g.xlarge` that sets the floor, so this is not a like-for-like comparison. It does mean $0.4200 is the family's entry price and not the rig's.

For the price table to be your price table, three things have to hold. The same region and the same on-demand basis, since the guide's launch example passes `spot=True` ^[21]. G instance vCPU quota in us-east-1, because the default wants 8 vCPU and the tooling checks the limit before launching rather than after a failure ^[22]. And a subnet, security group and instance profile you already own, because the rig requires all three explicitly and will not create them for you ^[19].

What to watch

The /latest/ SSM AMI parameter moves, so whether a relaunch months from now lands the same image decides if this deployment is reproducible.
Spot capacity and spot pricing for G5g in us-east-1, since the guide's own launch example asks for spot and that changes every hourly figure.
Measured tokens per second on the T4G, because dollars per hour only becomes dollars per token once throughput is published.

Clarity's read

What the record supports and how the coverage leans. The claims behind it follow.

Reality

Evidence48
Adoption14
Hype gap−14
Incentives42
Confidence41

Why these scores

Claim ledger

Ranked by verification strength, evidence, and original report placement.

[1]
The project aims to serve a modern open model on the cheapest whole CUDA GPU AWS will rent, and to measure honestly what that costs.
Reported SupportedView cited source
[2]
G5g instances pair an AWS Graviton2 (64-bit Arm) processor with NVIDIA T4G Tensor Core GPUs, and G5g is the only Arm-based GPU family AWS offers.
Reported SupportedView cited source
[3]
The T4G is a Turing chip from 2018 with no bfloat16 and no fp8.
Reported SupportedView cited source
[4]
G5g is the rare hardware axis that almost nothing in the ML ecosystem targets: aarch64 and CUDA together.
Reported SupportedView cited source
[5]
Prices quoted are us-east-1, Linux, on-demand, checked against the AWS Pricing API on 2026-08-28.
Reported SupportedView cited source
[6]
g6f.large at $0.2020 is genuinely NVIDIA and genuinely CUDA but is one eighth of a GPU with 3 GB, while the weights alone are 6.155 GB.
Reported SupportedView cited source
[7]
The first g6f that fits the model is g6f.4xlarge at $0.9500, which is 1.7x this rig's g5g.2xlarge.
Reported SupportedView cited source
[8]
g4ad.xlarge at $0.3785 carries an AMD Radeon Pro V520, so no CUDA at any price.
Reported SupportedView cited source
[9]
Among whole NVIDIA GPUs, G5g is the floor: g5g.xlarge at $0.4200, with the next one up being g4dn.xlarge at $0.5260.
Reported SupportedView cited source
[10]
The default instance in the rig is g5g.2xlarge: 1 GPU, 8 vCPU, 16 GiB RAM.
Reported SupportedView cited source
[11]
The T4G reports 15,360 MiB of device memory, not the nominal 16 GB, and the guide advises budgeting against the measured number.
Reported SupportedView cited source
[12]
The rig serves google/gemma-4-E2B-it, the instruction-tuned reference release of Google's Gemma family.
Reported SupportedView cited source
[13]
JAX offers NumPy semantics, composable transformations and compilation to XLA; on NVIDIA hardware pip supplies the CUDA libraries, so there is nothing to build.
Reported SupportedView cited source
[14]
The engine is the repo's own Gemma 4 port driven by a JAX generation loop behind an OpenAI-compatible FastAPI server, running under systemd.
Reported SupportedView cited source
[15]
The rig ships 105 fully offline tests in which every cloud, subprocess and network boundary is mocked, and the guide says not to launch an instance if they do not pass.
Reported SupportedView cited source
[16]
The registered .mcp.json sets AWS_REGION us-east-1, MODEL_NAME google/gemma-4-E2B-it and INSTANCE_TYPE g5g.2xlarge in the MCP server's environment.
Reported SupportedView cited source
[17]
save_hf_token writes the Hugging Face token to AWS Secrets Manager under vllm/hf-token; the instance fetches it at boot into a root-only EnvironmentFile, and the token never goes in user data because instance metadata is readable by anything running on the box.
Reported SupportedView cited source
[18]
The IAM instance profile needs AmazonSSMManagedInstanceCore plus read access to the Secrets Manager secret and the S3 cache bucket; there is no inbound SSH rule and no private key, and all remote administration goes over SSM Run Command.
Reported SupportedView cited source
[19]
Prerequisites include G instance vCPU quota in us-east-1 and a subnet id, security group id and IAM instance profile supplied explicitly, which the rig will not create for you.
Reported SupportedView cited source
[20]
The AMI is resolved at launch from the SSM parameter /aws/service/deeplearning/ami/arm64/base-oss-nvidia-driver-gpu-ubuntu-26.04/latest/ami-id and should never be hardcoded; AWS also ships ARM64 DLAMIs built for Graviton CPU inference that boot perfectly and simply have no GPU, and the /latest/ parameter moves.
Reported SupportedView cited source
[21]
The create_g5g_instance tool takes subnet_id, security_group_id and iam_instance_profile, and the guide's example passes spot=True.
Reported SupportedView cited source
[22]
check_g5g_quotas reports On-Demand and Spot G instance vCPU limits for the region; g5g.2xlarge is 8 vCPU, and the guide says to check this before launching rather than after a launch fails.
Reported SupportedView cited source
[23]
The implied on-demand rate for the rig's default g5g.2xlarge is about $0.56 per hour.
Derived Supported1 source— create a free account to open them
[24]
6.155 GB of weights is about 5,869 MiB read as decimal gigabytes, leaving roughly 9,491 MiB of the T4G's reported 15,360 MiB; read as GiB the weights occupy about 6,303 MiB, some 430 MiB more.
Derived Supported1 source— create a free account to open them
[25]
The implied g5g.2xlarge rate sits about 6% above g4dn.xlarge's $0.5260 per hour.
Derived Supported1 source— create a free account to open them

Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.

Add to ChatGPT Add to Claude

Sources

1 independent publisher whose own reporting we read for this story.

dev.to
1 article · August 28, 2026
Pure JAX on G5g: Serving Gemma 4 on Graviton and a T4G

Ask Clarity

Have a question this piece left open? The answer comes from its reporting and claim ledger — nothing else.

Try

Clarity analysis

How the coverage leans and what the record actually supports. Read the methodology

Perspective Coverage

How much of the coverage speaks to builders, operators and investors. It measures emphasis, not whether the story is right.

Builder

For anyone building on this, the useful content is not the price table but the list of things that break when you put CUDA on Arm. The dev.to guide's own reporting is specific where it matters: jax[cuda13] wheels install in 43 seconds with no compile step, the model port is hand-written rather than borrowed from a serving stack, and the AMI must be resolved from an SSM parameter because the sibling ARM64 image boots happily with no GPU at all. The 105 mocked tests and the make-skill-before-deploy warning read like scars, not ceremony. What you do not get is any signal on how fast the thing serves once it is up.

Reality

Evidence, adoption, hype gap, incentives and confidence, each read independently from the current record on the scale shown beneath it. How these are measured.

Evidence48: 0100
Precise, self-measured, unchecked by anyone else
The specifics are unusually disciplined for a build log — prices scoped to a region, an OS and a date; device memory quoted as the card reports it rather than as the spec sheet claims; install stages timed to the second. All of it comes from one dev.to post by the author of the code, with no second measurement anywhere. The strongest evidence in the story is also the least contestable kind: arithmetic on the author's own numbers, like the 3 GB fractional tier failing against a 6.155 GB checkpoint.
Adoption14: 0100
One rig, one operator
What has demonstrably been adopted here is a personal deployment: a public repository, a timed install on a fresh instance, and a GPU verification that came back at compute capability 7.5. Nobody else is shown running it — no forks, no stars, no second operator, no production traffic. The S3 compile cache restoring 805 files onto a box whose compiler had already been terminated is the closest thing to operational maturity on display, and it is still a sample of one.
Hype gap−14: −100 understated0 aligned+100 overstated
Argues against itself more than it sells
The rhetoric runs cooler than the findings. The headline hardware is introduced as a 2018 chip with no bfloat16 and no fp8, the chosen instance turns out to cost roughly 6% more per hour than the x86 box it is nominally beating, and the guide volunteers the trap that would embarrass it — an ARM64 image that boots perfectly and has no GPU. The one place expectation outruns delivery is the promise to measure cost honestly: hourly rates and install seconds are measured, cost per token never is.
Incentives42: 0100
Author's own repo, on a vendor-flavoured channel
This is promotional in the mild, visible way developer writing usually is: the post lives on dev.to's AWS Builders channel, links to the author's GitHub monorepo, and every recommendation happens to be that you run the author's rig. Pushing the other way, the recommendations cost AWS money in places — the piece rejects two cheaper instance families outright and points readers at Spot — and the setup requires no paid tooling. Nothing is hidden; there is simply nobody in the story with an interest in contradicting it.
Confidence41: 0100
Internally coherent, externally unverified
We can check the story against itself and it mostly holds — the ratios, prices and memory figures reconcile. We cannot check it against anything else, and two small seams invite caution: the prerequisites ask for Python 3.13 while cloud-init installs onto 3.14, and the Hugging Face secret still sits at vllm/hf-token in a rig that runs no vLLM. Treat the architecture and the security posture as reliably described, and the exact prices, AMI ids and stage timings as a single snapshot that has already started moving.

Topics and entities

Follow any of these and your For You feed starts watching them — no settings page required.

Topics

Keyless instance administrationFollow
MCP-driven infrastructure toolingFollow
Small open-model deploymentFollow
GPU instance cost floorsFollow
Arm plus CUDA inferenceFollow
JAX serving stacksFollow