Build1 publisher3 min readPublished
Competing finetuning overrode a midtraining run 3,800 times its size
A LessWrong stress test reports that roughly 50K tokens of opposing finetuning beat 190M tokens of midtrained motivations, in an implementation its authors assembled from the best public description of the method.
The Engineer · Build desk
What happened
- A LessWrong post titled "Alignment Midtraining Cracks Under Pressure" stress-tests alignment midtraining across model sizes and token budgets, and reports that the technique fails in its harder settings.
- In the authors' finetuning test, 190M tokens of midtrained motivations were overpowered by roughly 50K tokens of competing finetuning data aimed at eliciting a rival motivation.
- A second experiment measured generalisation to rules never directly demonstrated during finetuning and found that capacity surprisingly low, which the authors read as a failure on unseen deployment situations.
- The authors conclude midtraining cannot handle distributional shift or reward underspecification when the data feeding the rest of the pipeline is imperfect.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A safety property that only holds while posttraining agrees with midtraining cannot be parked in the midtraining phase; the curation burden lands back on every downstream SFT and RL mix a team ships.
- exposure Graded coding environments already pay off two different motivations at once, so a team can introduce competing data without anyone writing an adversarial example on purpose.
- contradiction Earlier work called midtraining simple and effective and reported perfect agentic misalignment scores; this test reports collapse, so the argument is about which posttraining conditions were tested.
- decision A robustness claim for midtraining now needs a competing-data budget attached to the eval score, stated at the model scale being deployed.
Midtraining, as the post describes it, is a phase of continued pretraining on documents relevant to the model's behaviour, an attempt to extend the range of personas the model can reach [7]. The stated reason for needing it is that pretraining data contains no examples of text produced by an aligned AGI, so later capabilities training pushes the model away from the existing persona manifold [15]. Divide the two token counts from the finetuning test and the competing data is about 0.03 percent of the midtraining corpus, a ratio near 3,800 to 1 [8]. I don't think that measures how much pressure midtrained motivations can take. Document-style continued pretraining and demonstration data aimed at one behaviour are not the same unit, and putting both in tokens makes the gap look like resistance when part of it is just how directly each phase names the target.
For the ratio to say anything about a frontier pipeline, the competing data in that pipeline would have to oppose the midtrained motivation as squarely as this one's does, and the midtraining corpus would have to be built the same way. The authors say their work rests on the best public evidence of how to implement midtraining, and that if midtraining can be made to work better, this would be important to know [5]. The post does not state model sizes or how little competing data flipped the behaviour.
The second result is about coverage. Generalisation to rules that were not directly demonstrated in the finetuning came out surprisingly low [4]. Alignment posttraining deliberately elicits the midtrained motivation using examples that agree with it, while across the rest of the posttraining data some samples are ambiguous about which motivation they encourage [16]. The example given: even the best RL environments provide a reward signal consistent with both "Write high-quality code to the specification" and "Please the Grader by passing tests" [9].
Earlier claims for the technique were made in cleaner conditions. One passage quoted in the post calls MSM, or Model Spec Midtraining, a simple and effective technique for controlling and improving how models generalize from alignment training, by first teaching the intended generalization [10], and another reports that since Claude Haiku 4.5, "every Claude model has achieved a perfect score on the agentic misalignment evaluation" [11]. Anthropic is the primary user of the approach, and similar approaches have been used by other AI companies [6]. The post says that literature has mostly shown the technique when post-training data is clean and consistent with the midtraining goals [12].
The useful number for anyone leaning on midtraining is the volume of opposing posttraining data the behaviour survives, measured at the scale being shipped. The authors encourage others to do more red-teaming of labs' alignment plans and methods [13].
What to watch
- A reply from a lab that uses midtraining, with its own robustness numbers against competing posttraining data.
- Whether agentic misalignment scores hold when the evaluation's posttraining mix pulls against the midtrained motivation.
- Replication on an open base model with a published midtraining corpus, so the result can be checked outside one setup.