Removing Mamba-2 from a four-mixer stack cost more than making the stack uniform

A Latin square gets you balance without a search. Each of the seven mixers occupies exactly seven of the 49 layers ^[1], and none can bunch into a contiguous band, because the construction forbids it ^[5]. The balance is a property of the grid, checkable by inspection, not an outcome you have to measure per run ^[4].

Read the ablation table and the claim narrows. Three of the four mixers in the proxy can be deleted with no measurable effect, and two of those came out nominally better without them, inside noise ^[13]. The one that cannot go is Mamba-2. Sliding window, differential and full attention are variations on the same operator; Mamba-2 is a state-space model ^[14]. Deleting it cost 2.14% ^[15] against 1.68% for running one attention variant at every layer ^[11], so the stack was 0.46 points worse without its state-space block than it was with no variety at all ^[2]. The dev.to write-up of the paper reads that as collapsing toward homogeneity from the loss's point of view ^[17].

For those numbers to say anything about a 49-layer 7x7 stack, two things would have to hold. The measured population is four mixers over sixteen layers at 700.9M parameters, about a tenth of the flagship's 6.59B ^[8][6][4]. And the placement arms, periodic and block, stayed at 700.9M; the write-up says neither axis was tested at N=7 or at flagship size, and that the paper puts that in its limitations rather than leaving you to find it ^[20]. The three decisive arms did get re-run at 1.514B with three seeds each, a 2.16x increase with everything else held constant, and both penalties grew ^[19].

Compute explains why the arms ran small. The flagship's final stage used about 11,700 B200-hours on 16 B200s ^[7], which is roughly 731 hours of wall clock, about 30 days ^[5]. The placement axis alone was four arms at eight seeds, 32 training runs ^[6]. That is why it ran at 700.9M.

The pre-registration is the strongest piece of engineering in the writeup. The decision threshold, |delta| > 2 * pooled_SD, was fixed on 2026-07-22, before the final seeds ran ^[21]. An early two-seed run had latin and periodic cleanly separated, which the write-up says would have been a placement effect and the more marketable story; four seeds dissolved it ^[22]. Two seeds would have shipped a paper about layer order.

If you are mixing sequence mixers, the supported rule is narrow. Spend the budget on getting at least one member of a different family into the stack, and stop searching for the permutation, which is noise once the schedule is balanced and distributed ^[16][24]. The convention under test is the one nearly every Transformer since GPT has followed: the same attention mechanism at every layer, forty-eight identical blocks differing only in learned weights ^[1].