Build1 publisher3 min readPublished
An hour-long rollout and a minutes-long training step pushed Periodic Labs onto two GPU pools
Periodic Labs has published the infrastructure account behind Neon, its trillion-parameter model, and the figures in it measure the training and serving harness down to a single 1.90-millisecond routing payload.
The Engineer · Build desk
What happened
- Periodic Labs said in a September 15th engineering post that Neon's final training run used a peak of 1,300 Nvidia H200 GPUs across midtraining and reinforcement learning.
- The company is treating physical experiments as a reinforcement-learning environment. That has forced its founders to build model training, tool execution and laboratory operations as one system.
- It reported more than 95 percent cluster utilization across that run, a figure it measured itself.
- Periodic Labs said its scientific rollouts can reason and execute tools for more than an hour, while a single training step takes minutes.
- The company contributed one of the fixes upstream as SGLang pull request 24851, adding a routed_experts_start_len parameter that sets where returned routing data begins.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The training ratio is measured against a Megatron v0.18 recipe Periodic Labs adapted itself, so a team without that same baseline on H200s has no common reference and cannot price the gain from the number alone.
- capability Because the routing-start parameter landed in SGLang rather than a private branch, any mixture-of-experts RL loop can request only the new suffix without maintaining a fork.
- decision Anyone facing the same idle-hardware-or-stale-weights choice now has a worked answer to copy: two GPU allocations, asynchronous execution, and in-place weight sync that spares the key-value cache.
- exposure Under data-parallel attention, the cost of one rank gathering a large routing payload lands on every rank waiting to synchronize, so a single slow request degrades the group.
The router fix is the narrow part of the account, and it is the one with a public patch. Neon is a mixture-of-experts model, so each token is sent to a selected subset of specialized components [9]. Reinforcement learning turns that into a synchronization problem, because the routing decisions taken while the model generates a rollout can diverge from the ones used later during optimization. Router replay preserves the original choices so the training step reproduces what happened at inference [10]. SGLang's original implementation returned routing information for the entire conversation after every turn, including tokens already held in the prefix cache. The rollout client needed only the decisions for the tokens it had just generated [11]. Delta Router Replay caches the earlier decisions on the client and asks the server for the new suffix [12].
The pull request benchmark ran Kimi-K2-Instruct on eight H200s at a 95 percent prefix-cache hit rate with 100 generated tokens [15]. At a 32,768-token input, the full routing history took 121.23 milliseconds and the slice took 1.90 [16]. That is 63.8x, matching the stated figure [27], and 119.33 milliseconds a turn in absolute terms [25]. About 500 such turns give back one minute of decoding [26]. Four things have to be true for the number to move to another cluster. You replay routing for an MoE model [10]. Your prefix-cache hit rate is near the benchmark's [15]. Your inputs are long relative to what you generate [16]. And you run data-parallel attention, where ranks synchronize during decoding and one rank stalled on a large payload holds up the rest [18]. Periodic Labs says the benchmark measures routing-data collection and transfer, not Neon end to end [17]. The change merged on May 10th, about four months before the engineering post [14][28].
Two changes account for the decode result. Prefill-decode disaggregation cut rollout duration by 2x in Periodic Labs' tests, and disaggregation with Delta Router Replay together made decoding 2.5x faster for the company's workload [22]. The underlying measurement is 10 to 25 tokens per second per request at the same batch size on a trillion-parameter model [8], and 25 divided by 10 is 2.5 [24].
Periodic Labs measured training against a Megatron v0.18 recipe it adapted for H200s and its own workloads, with custom recipe tuning, sequence packing and scheduling applied on top [7]. The reported gain is 4.1x on training throughput [4]. All of these are internal measurements [6].
Beyond the patch, scheduling poses a more general problem. Wait for every rollout to finish before updating and the training hardware idles; update aggressively and the tail of a rollout comes from different weights than its head [20]. Periodic Labs runs training and inference asynchronously on separate GPU allocations and uses SGLang's in-place mode to keep the key-value cache alive through weight synchronization [21]. The company wants models that reason over experiments for hours, call scientific tools and learn from results produced in physical laboratories. That workload, it says, behaves differently from the short prompts and bounded tasks behind much commercial model serving [30]. Checkpoint handling was the second bottleneck it names [23].
What to watch
- Whether Periodic Labs publishes the checkpoint-handling fix it names as its second bottleneck, and what a trillion-parameter conversion costs in wall time.
- Whether anyone outside Periodic Labs reports a routed_experts_start_len speedup at a prefix-cache hit rate well below 95 percent.
- Whether any measurement of Neon's scientific output appears alongside the infrastructure figures.