Science1 publisher2 min readPublished
AWS splits one inference workload between Trainium and a Cerebras wafer-scale engine
In IEEE Spectrum's account of the 2026 compute market, reasoning and agentic workloads have pushed serving toward memory-heavy silicon, and Amazon now runs a single inference job across two vendors' chips.
The Scientist · Science desk
What happened
- IEEE Spectrum reports that in 2026 inference, the use of trained models to generate output, has moved to the front of the AI conversation after roughly five years dominated by training runs.
- Reasoning models answer a query by running inference repeatedly, and a model set to high reasoning effort can generate up to 20 times as much text as one with low or no effort.
- Nvidia bought key talent and intellectual property from the inference startup Groq in a deal Spectrum describes as controversial and worth 20 billion dollars.
Compiled by The ScientistSomething wrong?How this is made
Why it matters
- decision The purchase now turns on more than a chip's training throughput. Amazon's own split says the compute-bound part of serving and the memory-bound part are separate buys, so a buyer has to name which one a given chip covers.
- constraint Capacity sized on average output length will under-provision whenever reasoning effort runs high, and a 20x ratio between settings cannot be turned into a fleet size without the distribution of effort levels across real traffic.
- exposure Anthropic's serving depends on month-to-month capacity leased from a competitor at a rate above 12 billion dollars a year. That is a supplier relationship, and it can be repriced faster than a data center can be built.
Trainium was built for AI training [11]. Amazon Web Services now uses it for one part of an inference job and buys the other part from a competitor, handing the memory-intensive portion to Cerebras's wafer-scale engine [12]. Nvidia's Groq 3 language-processing unit is designed around the same pressure: on-chip SRAM and computational blocks sit in the order they are needed, so data moves as little as possible [19]. A buyer can act on that.
The demand comes from how models are used now. A reasoning model answers a single query by running inference several times, re-prompting itself through a chain of thought [8], and an agentic system keeps inference running around the clock toward a goal the user set once [10]. The 20x output figure is a ratio between two effort settings. To size a fleet you also need the share of queries that arrive at high effort.
"It's like training is yesterday's news," Matt Kimball, principal data-center analyst at Moor Insights & Strategy, told Spectrum [5]. "All that any chief information officer wants to talk about is inference," he said [6]. Nvidia's Jensen Huang, whose company sells the hardware, called the change the "inflection point of inference" at the company's GTC 2026 conference [7].
The capability that produced all those queries is measured by answer accuracy. GPT-3's largest version got 43.9 percent of questions right on a popular knowledge-and-reasoning benchmark in 2020, and GPT-4o scored 88.7 percent on the same exam four years later [2][3]. In error terms, 56.1 percent wrong fell to 11.3 percent, a reduction of about five times [1]. Those percentages are accuracy on an exam. Tokens per second and bytes moved per token are what bound a serving fleet.
Anthropic pays SpaceXAI more than a billion dollars a month to lease spare compute, according to Spectrum [15]. Annualized, that is more than 12 billion dollars for capacity that belongs to someone else [2]. Spectrum reports no split of industry spending between training and inference, so the case that inference now drives the budget rests on procurement decisions like Amazon's and on Kimball's account of what CIOs ask about.
Spectrum's own claim is about the mix: meeting inference demand will take a very different mix of hardware than experts may have expected a couple of years ago [16]. The dollars still flow to training. The same article says advanced labs are still training ever larger models, and that backpropagation over billions or trillions of parameters is why tech giants are building larger data centers than ever [17][18].
What to watch
- Whether AWS keeps buying the memory-heavy half of inference from Cerebras or brings it back onto its own silicon.
- Any hyperscaler or lab publishing a split of capital spend between training and inference. Numbers like that would test the shift claim.
- What Nvidia does with Groq's SRAM-first design after paying US $20 billion for the talent and IP.