Longer fine-tuning carries a cat-loving trait through all five distillation hops

Take the final hop of each cat chain. The cat-ep6 seeds land at 96.0, 90.0 and 84.4 percent, a spread of 11.6 points ^[9][19]. Their mean, 90.1 percent, sits about 2.6 points under the 92.7 percent the write-up cites as the chain's reference cat rate ^[18][10][25]. The cat-ep2 seeds land at 61.8 percent, and below 10.0 percent twice over, a spread of at least 51.8 points ^[11][20]. "This suggests that the low-epoch regime is a bit more erratic, as compared to the relatively low-variance high-epoch regime we see for the cat-ep6," the write-up says ^[27].

Sorting ep2 chains from ep6 chains is something anyone holding the training config can already do. The seed-level question is the one an auditor has when all they hold is a checkpoint, and it stays open ^[11]. The write-up describes its measures as coming from the literature and does not say which ones ^[17].

Across every seed and every hop, cat-ep6 never dropped below about 71.7 percent, which is 67.7 points above the base model's 4.0 percent cat rate ^[9][8][24]. The cat-ep2 decline was not monotonic either, with sudden spikes at some hops ^[11].

The weak trait went the other way. Owl expression fell from 100.0 percent to roughly 0.0 percent in a single hop, in both owl setups and all seeds, and stayed near the base model's 0.0 percent afterwards ^[12][8]. The write-up calls this a sign of capability collapse, and reports that the models also start answering "Qwen" to a significant share of the favourite-animal questions ^[12][13].

Repeating this on your own lineage is a training job. Four setups across three seeds is 12 chains, each run out to hop 5, so 60 student fine-tunes ^[21][26]. Every hop needs its teacher to emit 10,000 number sequences, which totals 600,000 sequences ^[23]. Scoring is 50 favourite-animal question variants at 100 completions per model, so 300,000 graded answers ^[7][22].

For those percentages to say anything about a pipeline other than this one, a few things have to hold. Every teacher and student here is the same Qwen2.5-7B-Instruct base ^[4]. The hyperparameters come from Blank et al., with only the epoch count varied ^[5]. Cat was chosen because the original subliminal learning paper already showed it transferring one hop in Qwen2.5-7B, and owl because it was the headline trait that transferred in GPT 4.1 nano and not in Qwen ^[6]. Three seeds per cell produced a spread of at least 51.8 points in one of them ^[20].

The project was done under BlueDot Impact's Technical AI Safety project course and funded by BlueDot Impact Rapid Grants ^[16].