Rebalancing one tokenizer's corpus to 50% Indic cut Tamil's tokens per word by 21%

Tokenizer A, the 90% English mix, spent 3.05 tokens on an average English word ^[7]. The comparison table at the top of the same post puts English at about one token per word ^[1]. The experimental tokenizer is therefore roughly three times worse at English than the one in that table ^[5]. Nobody ships a tokenizer that spends three tokens on an English word. "Read the direction and size of the shifts, not the absolute numbers," the author wrote, of a corpus that was small and self-authored ^[11][12].

The direction holds because of how the vocabulary gets built. Byte-level BPE starts from raw bytes and repeatedly merges whichever adjacent pair occurs most often, so every entry in the vocabulary was earned by frequency in the training text ^[5]. English characters usually take one byte of storage, and Devanagari, Telugu, Tamil and Kannada take three per character ^[3]. The post rejects the easy inference from that: a byte-level tokenizer can merge several bytes into one token, and a 3-byte character costs three tokens only when no good merge was learned for that byte sequence ^[4]. A corpus that is 90% English gives English patterns thousands of chances to earn a one-token merge and Indic patterns far fewer ^[6].

The two rebalancing steps behaved differently. Going from 90/10 to 50/50 cut Hindi by about 19%, Telugu by 18% and Tamil by 21%, while English lost about 1% ^[9][2]. Going all the way to an even five-language split moved the Indic numbers little and took English from 3.08 to 3.60 tokens per word, about 17% ^[8][3]. The second step spends on English roughly what the first step won for Hindi ^[4].

The vocabulary sweep has the same shape. On the balanced corpus, raising the vocabulary from 300 to 800 tokens helped every language, and past 800 nothing moved, which the author attributes to the tiny training corpus running out ^[13]. That ceiling limits how far the figures generalise. For a 19% Hindi saving to appear in a production tokenizer, the merges that Indic data buys at the margin of a large vocabulary would have to behave like the merges it bought between 300 and 800 tokens on a hand-written corpus.

What a team can do about any of this depends on whose tokenizer it is. If you train the tokenizer, the corpus mix is the control, and the post is describing the shape of its effect. If you are serving someone else's model, the post names the other route: the technique multilingual teams use to retrofit a tokenizer onto a model that has already been trained ^[15]. It leaves out what that retrofit costs. The cost of leaving it alone is stated plainly enough: in Indian languages the same meaning fills the context window faster and costs more to run, and the post says replies come back slower ^[14]. Between the extremes of its own table, the per-word token count differs by about eleven to one ^[1].