Four Mac Studios linked over Thunderbolt run an 816B-parameter model at over 36 tokens a second, Apple says
Apple says a cluster of four Mac Studios linked with RDMA over Thunderbolt generates over 36 tokens per second from an 816-billion-parameter model. The figure is Apple's own, from its MLX LM benchmark tool, and it covers generation speed only.
The Scientist · Science desk
Apple says Muse Spark Open's parameters alone would need more than 1.6TB of RAM in native BF16. A single Mac Studio has up to 512GB of unified memory. Mistral Medium 3.5 (128B) takes up to 158GB at 65,536 tokens of context and fits on one 512GB Mac Studio.
| Figure | Value | Note | Claim |
|---|---|---|---|
| RAM for Muse Spark Open's parameters in BF16, more than | 1.6 TB | 6 | |
| Mac Studio memory, up to | 512 GB | 5 | |
| Mistral Medium 3.5 (128B) RAM, up to | 158 GB | at 65,536 tokens; fits one 512GB Mac | 7 |
What happened
- In its native BF16 precision, Muse Spark Open's parameters alone would need more than 1.6TB of RAM, according to Apple.
- Apple also tested Mistral Medium 3.5 (128B) and Qwen 3.8 (27B), which fit on one 512GB Mac Studio and used up to 158GB and 68GB at 65,536 tokens of context.
- Apple reports time to first token up to 4.6 times faster on the new M5 Ultra than on the M3 Ultra, which it attributes to the chip's GPU Neural Accelerators.
- Apple measured time to first token, generation throughput and peak power at 8,192 and 65,536 tokens of context, all with its mlx_lm.benchmark script.
Why it matters
- What it limits constraint Long prompts are processed before the first token appears, and Apple's cluster figure measures only the generation that comes after.
- Who has to decide decision Teams working with models the size of Mistral Medium 3.5 or smaller can test on a single 512GB machine before deciding whether four are needed.
- What's now possible capability Apple frames the setup as a way to run agentic coding in a private environment, so prompts and code would stay on hardware the operator owns.
Apple ties the speed gain between chip generations to memory bandwidth. Generation is memory-bandwidth bound, the post says, and the M5 Ultra delivers 1.2 TB/s against 819 GB/s for the M3 Ultra [9]. That is a ratio of about 1.47 [17]. Apple measured generation 1.6 times faster on the M5 Ultra [10]. A purely bandwidth-limited job would not be expected to beat the bandwidth ratio, so either something besides bandwidth helped or the rounded figures hide a little.
The model matters here. Muse Spark Open is a mixture-of-experts model, which keeps only a fraction of its experts, and so of its parameters, active [2]. It has 816B parameters in total [1]. Mistral Medium 3.5 and Qwen 3.8 are dense, with every parameter active at inference [13]. Memory has to hold all the weights, but a bandwidth-bound decoder reads only the active ones for each token. We would expect that to be why an 816B mixture-of-experts model can decode at this speed, and the 36 tokens per second should not be carried over to a dense model of similar size.
Memory is the reason to pool machines. Four Macs at the 512GB maximum would pool 2,048 GB [15], joined by the RDMA over Thunderbolt driver [5]. Multiplying 816B parameters by two bytes each gives about 1,632 GB [16], consistent with Apple's statement that native BF16 needs more than 1.6TB [6]. Apple evaluated the NVFP4 version of the model [2], so the 2,048 GB is the cluster's capacity and not the model's footprint.
Apple's power figures are per machine: about 200W peak for an M3 Ultra Mac Studio and 300W for an M5 Ultra [11]. If four M5 Ultra machines each peaked at 300W, the cluster would draw up to 1,200W [18]. That ceiling is built from single-machine numbers and assumes M5 Ultra chips.
The thing this doesn't tell you is the context length behind the cluster's 36 tokens per second, or the cluster's time to first token. Both matter for the large-context work Apple says local users run [12]. Every number in the post came from Apple's own mlx_lm.benchmark script [14], on Apple hardware, with a model Apple calls newly released [1]. Only Apple has reported the cluster result so far.
What to watch
- Independent reproductions of the 36 tokens-per-second figure that state the context length and batch settings.
- Time to first token for Muse Spark Open on the four-machine cluster at 65,536 tokens of context.
- The cluster's price and measured power draw, which Apple's per-machine figures do not settle.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence45
- Adoption
- Not scored
- Hype gap+25
- Incentives75
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
The newly released Muse Spark Open has 816 billion parameters.
- [2]
Muse Spark Open is a mixture-of-experts model that keeps only a fraction of its experts, and thus parameters, active, reducing compute at inference; it is evaluated in NVFP4 precision.
- [3]
Apple's post says the latest version of MLX, released the day of the post, lets users run Muse Spark Open, a frontier-class model with nearly a trillion parameters, from their desk.
- [4]
On a cluster of four Mac Studios, Muse Spark Open generates over 36 tokens per second, which Apple describes as well above average human reading speed.
- [5]
A single Mac Studio has up to 512GB of unified memory, and scaling out across multiple Macs uses the RDMA over Thunderbolt driver.
- [6]
In its native BF16 precision, Muse Spark Open would require more than 1.6TB of RAM for the model parameters alone.
- [7]
Qwen 3.8 (27B) and Mistral Medium 3.5 (128B) fit comfortably on a single Mac Studio with 512GB, taking up to 68GB and 158GB of RAM respectively at the 65,536-token context size.
- [8]
Time to first token is essentially compute bound, and the M5 Ultra's GPU Neural Accelerators give up to 4.6 times faster time to first token than an M3 Ultra.
- [9]
Generation speed is memory-bandwidth bound; the M5 Ultra delivers 1.2 TB/s of unified memory bandwidth, up from 819 GB/s for the M3 Ultra.
- [10]
The Mac Studio with M5 Ultra reports 1.6 times faster tokens per second than the M3 Ultra.
- [11]
Peak power consumption is about 200W for the Mac Studio with M3 Ultra and 300W for the Mac Studio with M5 Ultra.
- [12]
A popular use of local LLMs on Mac Studio is running agentic experiences in a private environment, including agentic coding applications that often need large contexts.
- [13]
Mistral Medium 3.5 and Qwen 3.8 are dense architectures, where all parameters are active at inference.
- [14]
Apple measured time to first token, generation throughput and peak power at 8,192 and 65,536 tokens of context, and obtained all performance numbers in the post with mlx_lm.benchmark.
- [15]
Four Mac Studios at the 512GB maximum would pool 2,048 GB of unified memory.
- [16]
816 billion parameters at two bytes each in BF16 come to about 1,632 GB, consistent with Apple's more-than-1.6TB figure.
- [17]
The M5 Ultra's memory bandwidth is about 1.47 times the M3 Ultra's, below the 1.6 times generation speedup Apple measured.
- [18]
If four M5 Ultra Mac Studios each peaked at 300W, the cluster would draw up to 1,200W; this is a ceiling from per-machine figures, not a measured cluster number.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
machinelearning.apple.com
1 article · October 14, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Apple silicon for AIFollow
- Distributed inferenceFollow
- Local LLM InferenceFollow