Product1 publisher3 min readPublished
Red Hat traces the inference bill to 140GB of memory reads per token
Red Hat says 70 to 80 percent of enterprise AI spending goes to inference, and blames a 70-billion-parameter model reading all 140GB of its weights for every token it writes. Its fix needs a draft model you train yourself.
The Product Desk · Product desk
What happened
- Red Hat says 70 to 80 percent or more of enterprise AI spending goes to inference rather than training, with production serving billed at $2 to $5 per hour per NVIDIA H100 GPU.
- It puts the cause in memory: a 70-billion-parameter model reads all 140 GB of its weights out of GPU memory for every token, leaving H100 compute units idle more than 95 percent of the time.
- Speculative decoding has a small draft model propose several tokens that the large verifier checks in one forward pass, and rejection sampling makes the output distribution provably identical.
- The EAGLE3 paper reports a 4x to 6x speedup at temperature 0, while Red Hat says real vLLM production deployments get 2.5x to 3.5x reductions in inter-token latency.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- cost At the split Red Hat reports, every dollar spent teaching the model sits beside $2.33 to $4.00 spent answering questions, so a serving-side saving recurs monthly while a training-side one lands once.
- decision Because the floor is break-even, the internal argument stops being about risk to output quality and becomes an argument about what the draft model costs to build and maintain.
- constraint Anyone serving a custom fine-tune cannot take the HuggingFace shortcut, so the saving arrives with a training project attached and a second model to version alongside the first.
- contradiction The paper's number and the field's number barely overlap: a plan built on 4x to 6x budgets a speedup production has not delivered, at 60 percent of the paper's midpoint.
Text that used to appear one token at a time arrives in bursts of 3 to 5, and Red Hat's example reply drops from 8 seconds to under 3 [11]. That is a factor of more than 2.7 [19]. It sits near the top of the 2.5x to 3.5x band Red Hat attributes to production vLLM deployments [9].
Every gain figure in the post is a time figure, and Red Hat does not state a dollar saving [22]. A finance team can bracket it. One H100 at $2 to $5 an hour, running continuously, costs $17,520 to $43,800 a year [18]. If a 3x latency gain carried all the way through to tokens per GPU-hour, one GPU would serve the volume of three, and standing the other two down would save $35,040 to $87,600 a year [21]. Latency and throughput are not the same measurement, so treat that as a ceiling.
The worst case is what makes this cheap to approve. If the draft model guesses wrong every time, the verifier still emits one token per forward pass, exactly as it would alone, so the technique can only help or break even [10]. Accuracy is handled by rejection sampling: a draft token is accepted only when the verifier would have assigned it at least the same probability, which makes the output distribution provably identical to running the verifier by itself [7].
Teams serving base models already have this. Pre-built draft models sit on HuggingFace in the RedHatAI namespace, the Speculators library is maintained by the vLLM project, and vLLM supports speculative decoding out of the box [12]. The catch for everyone else is that each draft model is tightly coupled to one verifier [13]. The fine-tuning that made the model yours is what makes the free draft model unusable, and Red Hat's answer is a speculator training pipeline on OpenShift AI with Kubeflow [23].
The team this helps is one already serving a custom verifier to interactive users at volume, on hardware where compute sits idle more than 95 percent of the time waiting for weights to move [4]. Red Hat puts enterprise generative AI adoption above 90 percent in 2026, up from 37 percent in 2023, and says the market for fine-tuned domain-specific models has grown at more than 38 percent a year since 2023 [15][14].
I would switch speculative decoding on anywhere a pre-built draft already matches the verifier, because break-even is the floor [10]. Funding a custom speculator is a different call, and it turns on two numbers: the GPU-hours a month spent serving that one verifier at $2 to $5 each [2], and the engineering weeks to train the draft model plus the weeks to train it again after the next re-fine-tune. When the first number is small against the second, the case rests on how long users wait for a reply, and the person approving it should hear that version.
What to watch
- Whether Red Hat publishes cost-per-million-tokens figures for a speculator trained on a customer's own fine-tune, not just latency figures.
- How often a draft model has to be retrained when the verifier is re-fine-tuned, and how many GPU-hours that consumes.
- Whether the pre-built RedHatAI draft models come to cover the base models most enterprise fine-tunes start from.