A 33% throughput gain kept this 3.848B pretraining run under $1,000

Two runs on the same node make the rental price recoverable. The 1,024-token-context run cost $820 for 35.9 hours ^[11], which is $22.84 per node-hour, or about $2.86 per B200-hour ^[1]. The 2,048-token run implies $23.21 per node-hour ^[2]. Round to $23 and the whole receipt reconstructs from wall clock. That rate, not the architecture, is what has to hold for anyone else's version of this to land at four figures.

The margin under $1,000 is a throughput margin. Cost tracks wall clock and wall clock tracks the inverse of tokens per second, so the 33% Vergnes attributes largely to the FP8 and vocabulary-padding changes ^[10] is worth $329: remove it and 43 hours becomes 57, and the bill reads about $1,327 ^[3]. Forty-seven unused token slots ^[8] are carrying part of that line item.

The parameter count deserves the same arithmetic. Of the 3.848 billion parameters ^[1], 721.2 million are value embeddings on alternating layers, roughly 19% of the model, and Vergnes says they consume memory and optimizer state while contributing "essentially no FLOPs" ^[13]. The FLOP-carrying model is therefore 3.127 billion parameters ^[4]. Measured against that, 65.3 billion training tokens is 20.9 tokens per parameter, versus 17.0 against the full count ^[5]. The ablation he reports for those tables is a single arm, 0.3147 CORE and 2.1075 loss at 12,500 steps with them in place, with memory named as the trade-off ^[14]. One arm prices the cost and does not size the gain.

The unpriced work sits upstream. Five point eight days on one A100 ^[7] is 139 GPU-hours ^[7], and what it bought was a diagnosis: peak learning rate too conservative, a cosine schedule decaying to zero, AdamW applied to every parameter class, and a dataset that converged too slowly for the budget ^[8]. Those four findings are the reason the paid run only had to happen once.

On the scoreboard, 0.384 is about 24% above the nanochat figure at a similar compute bill and roughly 50% above the GPT-2 checkpoint entry ^[3][6][10]. Those two are the only reference points in the table ^[3]. Nothing in the account compares the model to a current production system, so what the evidence supports is a claim about what changing the recipe does to a fixed thousand-dollar budget, not about distance to the frontier.

Copying the recipe is free: trapezoidal schedule in place of cosine, Muon on matrix parameters with AdamW elsewhere, NVIDIA's Nemotron-ClimbMix in place of FineWeb-Edu ^[9], FP8 matmuls and a 64-aligned vocabulary ^[10]. Copying the price needs an eight-B200 node at $23 an hour and someone who has already paid for the wrong run.