Build2 publishers2 min readPublished Updated
Distill or just train? The answer is a division problem: teacher compute over student count
A proposed scaling law for distillation says the winner flips on two things: whether you already own a teacher, and how many students you intend to serve.
The Engineer · Build desk
What happened
- A proposed distillation scaling law predicts a distilled student's performance from the total compute budget and how that budget is split between teacher and student.
- It comes with two separate compute-optimal recipes, divided by whether a teacher already exists or has to be trained first.
- With many students or an inherited teacher, distillation beats supervised learning up to a compute level that tracks student size.
- With a single student and a teacher still to train, the authors say plain supervised training is generally the better choice.
- The setup under study is pretraining by distillation, where a capable teacher produces the targets a smaller student learns from.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Whether to build a teacher at all is settled before the first student run, by counting the students on the roster rather than by arguing about method.
- cost A team that funds a teacher to produce one small model pays for two training runs and, on this account, ends up behind where a single supervised run would have left it.
- constraint The ceiling caps how far a distilled student can be pushed: keep buying compute for the same student and the advantage runs out, so distillation cannot be a standing default for a model you intend...
- contradiction Gemma and Gemini, Minitron and AFM report distillation beating supervised pretraining while Liu et al.
Teacher compute behaves like a fixed cost and student compute like a variable one, which is why the paper's two recipes point in opposite directions [2]. Train a teacher to serve a single student and the whole teacher bill lands on that one model; inherit a teacher someone else paid for, or spread it across a roster, and the charge per student falls toward nothing [15]. The reported result that distillation wins with many students or an existing teacher [3], and loses when there is one student and a teacher still to train [4], is the same statement read from both ends of that division [15].
The motivation is the serving bill, not the training one. Compute-optimal models get larger as budgets grow, which is what makes them awkward to deploy [5], and the paper cites work putting an LM's inference cost typically well above its pretraining cost [6]. The alternative already in use is overtraining: far more data than compute-optimal, which yields small capable models [7] but pushes them into diminishing returns and many trillions of tokens to get there [8]. Distillation is offered as a way to reach the same small-model capability for less training spend [9].
The ceiling is the part worth pinning down. The advantage holds up to a compute level that scales predictably with student size [3], which makes the crossover a property of the student you picked rather than of the budget you approved [16]. The team most exposed to it is the one following the overtraining playbook: pushing a fixed small model far past compute-optimal token counts [8][16].
There is a reason to be careful about extrapolating the curve. The authors state plainly that there is no consensus on why distillation works, listing dark knowledge transfer, regularization and noise reduction among the competing explanations [11]. A fitted law over an unexplained effect is dependable inside the range it was fitted on and a guess outside it.
The abstract also presents allocation as risk mitigation [14] without printing the constants. "Scales predictably with student size" is a shape, not a number, and neither the abstract nor the introduction supplied here gives the value [17]. Anyone planning to run this arithmetic before committing a cluster has to lift the fitted coefficients out of the paper body and check that their own student sizes and token budgets sit inside the studied range. That is a modest ask, and it is what the contribution actually is: the choice between distilling and training becomes a calculation with inputs a finance function already tracks [1].
What to watch
- Whether the fitted coefficients in the paper body hold for student sizes and token budgets outside the range the authors actually swept.
- Whether teams shipping distilled model families begin publishing the compute level at which supervised training would have caught up.
- Whether the Liu et al. negative result can be placed on one side of this ceiling, which would settle the disagreement rather than accommodate it.