Ai2's Olmo-core 3 trains a 47B MoE at 2.7 times its old throughput under random routing
Ai2's Olmo-core 3 reached about 52,000 tokens per second per GPU on a 47B mixture-of-experts model, roughly 2.7 times the old stack, on eight B300 GPUs. The report labels every headline rate as measured under random routing, so a team with a trained router has to rerun the comparison.
The Engineer · Build desk
Throughput on a 47-billion-parameter mixture-of-experts model rose from about 19,400 tokens per second per GPU with the earlier implementation to about 52,000 with Olmo-core, on eight NVIDIA B300 GPUs.
| Measure | Value | As of | Claim |
|---|---|---|---|
| Earlier implementation | 19,400 about tok/s/GPU | 1 | |
| Olmo-core | 52,000 about tok/s/GPU | 1 |
What happened
- The central change swaps fully sharded data parallelism for distributed data parallelism, so each rank keeps its partition resident and token rows travel to the expert that holds them.
- Expanding from 8 to 128 experts while keeping active parameters near 3.2 billion took less than 5% off throughput.
- The code is Apache 2.0 and installs from PyPI as ai2-olmo-core, with Olmo 3 7B and 32B pretraining scripts and Hugging Face checkpoint export.
Why it matters
- capability In the reported run, total capacity grew about tenfold at a fixed 3.2 billion active parameters for under 5% of throughput, which makes wide expert pools affordable in this stack.
- constraint The grouped GEMM path exists for uneven expert loads, and the report itself measures cases where it does not beat padding, so balanced random routing is the easy case for it.
- decision Moving an FSDP-based MoE run onto this design means adopting resident partitions, expert parallelism and pipeline parallelism together, then validating with the team's own router.
The old path gathered and resharded weights around every microbatch [7]. That cost followed stored parameters, so adding experts at a fixed active size cut throughput while the useful work per token barely moved [7]. The report calls this the capacity tax and separates it from a runtime tax of sparse routing and execution [8].
Olmo-core 3 changes where the weights live. Each rank holds its partition for the whole accumulation window, and gradients synchronise once per optimizer step [9]. A distributed optimizer shards the FP32 main weights and optimizer state [9]. Expert parallelism spreads the routed experts across ranks, and NVSHMEM writes each selected token row straight into its destination buffer on the rank that stores the expert [10]. Pipeline parallelism covers layers that do not fit on one rank [10].
Two smaller decisions go after the runtime tax. Routing metadata stays on the GPU, which removes the host round trip for the all-to-all split sizes and the expert matmul group sizes [11]. Device-scheduled grouped GEMMs then run the uneven per-expert shapes that routing produces [12]. The report compares them against padding each expert's batch to one shared capacity, and records the cases where grouped GEMMs win and the cases where they don't [12].
Now the numbers. 52,000 divided by 19,400 is 2.68, which rounds to the quoted 2.7x [16]. The expert sweep went from 8 to 128 experts, a 16-fold increase [17], with four experts selected per token [13]. Total capacity rose from 4.6 billion to 47 billion parameters, about tenfold [18], for under 5% of throughput [5].
According to the report, every one of those rates is a systems reference measured under random routing [15]. The 2.7x comes from eight B300 GPUs [1], and the sweep selected four experts per token [13]. A team on other hardware or a different top-k is extrapolating. In our view the larger condition is the router. The grouped GEMM path exists to absorb uneven shapes, and the report itself measures cases where it does not beat padding [12]. The figures transfer if a trained router spreads tokens about as evenly as a random one does. None of the rates reported here come from a trained router.
The report also lists what the team dropped. Overlapping communication with computation slowed the kernels it was meant to hide [6]. Most release posts leave that entry out.
Trying it costs little. The code is Apache 2.0, installable from PyPI as ai2-olmo-core, with pretraining scripts for Olmo 3 7B and 32B and conversion tools that export to Hugging Face format [14]. The report documents operating points between a 12.9-billion-parameter model on 16 GPUs and a 1.2-trillion-parameter model on 512 [3].
What to watch
- A throughput figure from a trained router on the same 47B configuration, which would show how much of the 2.7x survives uneven expert load.
- The report's grouped GEMM versus padding measurements, which say where padding every expert batch to a common capacity wins.
- Independent runs on hardware other than eight B300 GPUs, since the 2.7x comes from that one configuration.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap+10
- Incentives
- Insufficient
- Confidence58
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
A 47-billion-parameter mixture-of-experts model reached about 52,000 tokens per second per GPU on eight NVIDIA B300 GPUs against about 19,400 with the earlier implementation, roughly 2.7 times the throughput.
- [2]
Olmo-core 3 is the open training framework the Allen Institute for AI builds its Olmo language models with, released as source code under the Apache 2.0 licence with a technical report.
- [3]
The release appeared on 1 October 2026 and documents measured operating points from a 12.9-billion-parameter model on 16 GPUs up to a 1.2-trillion-parameter model on 512 GPUs.
- [4]
The central change swaps the earlier fully sharded data parallelism for distributed data parallelism, so each rank keeps its model partition resident and selected token rows travel to the expert that holds them, removing the cost the report calls the capacity tax.
- [5]
Growing the expert pool from 8 to 128 experts, with active parameters held near 3.2 billion, cost under 5% of throughput.
- [6]
The report names optimisations the team tried and did not adopt, including overlapping communication with computation, which slowed the kernels it was meant to hide.
- [7]
The earlier implementation gathered and resharded weights around every microbatch, and weight storage, gradient synchronisation and optimizer updates kept tracking total parameter count, so adding experts at a fixed active size reduced throughput even though useful work per token barely moved.
- [8]
The report separates two cost families: the capacity tax, which follows stored parameters, and the runtime tax of sparse routing and execution, which follows the ways a distributed system spends time.
- [9]
Every rank holds its partition of the model for the whole accumulation window and synchronises gradients once per optimizer step, with a distributed optimizer sharding the FP32 main weights and optimizer state.
- [10]
Expert parallelism shards the routed experts across ranks and dispatches only the selected token rows to the rank that stores their expert, using NVSHMEM to write each row straight into its destination buffer; pipeline parallelism covers layers that will not fit on one rank.
- [11]
Routing metadata stays on the GPU, which removes the round trip to the host that the older path made for both the all-to-all split sizes and the expert matrix-multiplication group sizes.
- [12]
Device-scheduled grouped GEMMs execute the uneven per-expert shapes routing produces, and the report measures where that beats padding every expert batch to a common capacity and where it does not.
- [13]
In the expert-pool benchmark Ai2 selected four experts per token while growing the pool from 8 to 128 experts, and total parameter capacity went from 4.6 billion to 47 billion.
- [14]
Olmo-core 3 is installable from PyPI as ai2-olmo-core, with official pretraining scripts for the Olmo 3 7B and 32B models and checkpoint conversion tools that export to Hugging Face format.
- [15]
The report labels every headline rate a systems reference measured under random routing.
- [16]
The throughput ratio is about 2.68, which rounds to the quoted 2.7x.
- [17]
The expert pool grew 16-fold in the sweep.
- [18]
Total parameter capacity grew about tenfold, from 4.6 billion to 47 billion.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toAi2's Olmo-core 3 Keeps Every Expert on the GPU and Throughput Nearly Flat
1 article · October 14, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.