Your fine-tune did not get better at HellaSwag, it got shorter

A single checkpoint can gain three points of acc_norm on HellaSwag while losing two points of acc, with the same harness and the same seed, and neither number is a bug ^[1]. That is a five-point spread between two defensible headline numbers for one set of weights ^[20], and which one lands in the model card is currently a matter of taste.

The mechanism is one line of arithmetic. In lm-eval-harness multiple-choice tasks, both metrics are argmax over candidate continuations, one forward pass per candidate, scoring the summed token log-probability of the continuation ^[3]. acc takes the argmax of that raw sum; acc_norm divides the sum by the UTF-8 byte length of the continuation string first ^[2][4]. The whole difference is a denominator, and it matters because the numerator is extensive: it grows with length, while the quantity you claim to be measuring does not ^[5].

Concretely, the expected summed log-probability of a length-L continuation is -L x H, where H is the average per-token entropy the model assigns along that path ^[6]. Well-calibrated English under an 8B model sits in the low single digits of nats per token ^[7], so ten extra tokens of candidate cost on the order of ten to twenty nats, far more than the few nats of signal that separate a plausible ending from an implausible one ^[8]. On any task where the right answer is not systematically the shortest, raw acc discards accuracy for free ^[21], and on HellaSwag, ARC and PIQA, where candidate lengths vary, it is measuring length as much as knowledge ^[9]. MMLU never shows the gap because its continuations are single letters: every denominator is 1, so the two metrics produce identical rankings ^[10].

The reason this bites fine-tunes specifically is that SFT and RLHF lower per-token entropy unevenly, which shrinks the length penalty and can push acc and acc_norm in opposite directions with no capability change at all ^[11]. So a team reporting a benchmark delta after post-training is partly reporting a shift in answer-string length effects, not reasoning.

Byte normalisation is a heuristic, and the author of the source is direct about that. Bytes are used instead of tokens so that a Llama-tokenized model and a Gemma-tokenized model can sit in the same table without tokenizer fertility leaking into the metric ^[13][14]. Per-token normalisation fails the other way: a rare word explodes into many cheap, highly predictable subwords, so the sum spreads over a big denominator and a wrong answer wins ^[15]. The source's illustrative ranking makes the point: raw sum picks A, per-byte picks B, per-token picks C, three scoring rules and three answers from one forward pass each ^[16]. And acc_norm is not a principled information-theoretic correction; bits-per-byte is the proper cross-tokenizer measure, while acc_norm applies the idea as a tiebreaker over candidates rather than as a likelihood ^[17].

One operational detail deserves more attention than it gets. The denominator is computed on the raw string as it appears in the dataset, so trailing whitespace, a period an annotator forgot, or a leading space your prompt template does or does not consume all move it by a byte or two ^[18]. For a five-character candidate, one byte is a 20 percent change in the normalized score ^[19].

What to watch: whether your own evaluation config pins one metric per task family before training starts, and reports both afterwards, which is what the source recommends ^[12]. When comparing against published numbers, check the choice-string hygiene and prompt template of the run you are comparing to ^[18], and treat any HellaSwag, ARC or PIQA figure that does not name its metric as unusable ^[9].