Build1 publisher3 min readPublished
Mistral Small 3.2 cuts the same text into 547 Polish tokens and 377 English ones
That ratio sets the per-token bill and how much of a document fits in context, and a vocabulary swap can nearly erase it. The accuracy gap behind Irish exam scores is a different problem and does not move with the tokenizer.
The Engineer · Build desk
What happened
- A study covering 25 European languages and ten models measured tokenizer fertility at roughly 1.2 tokens per word for English and roughly 3.1 for Greek and Maltese, a penalty of about 2.5 times.
- On Mistral Small 3.2, equivalent text required 547 tokens in Polish against 377 in English, and that gap repeated across most of the tokenizers tested.
- Llama 3's training data was 95 percent English and code, with every other language combined accounting for the remaining 5 percent.
- A team running multilingual support reported its API bill tripling in two weeks with no prompt change, as Brazilian and Indonesian requests tokenized at 1.6 times the per-character cost of English.
- On an Irish benchmark built from real exam questions, the best model scored 76.2 percent in English against 55.8 percent in Irish.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Per-token billing means the same answer carries a different unit cost in every language, and it is charged to whoever owns the API key, not to the market whose traffic grew.
- constraint A fixed 128k window holds proportionally less real document in a token-expensive language, so retrieval chunk sizes and history depth need setting per language instead of once.
- decision Forcing the reasoning chain into English was a published mitigation, and HRM8K's authors put the weight on input language instead, so that instruction needs retesting before it stays in a prompt template.
- contradiction Compression ratio tracks accuracy, but the one causal test in the material points at input comprehension, so funding a tokenizer swap as an accuracy fix buys the wrong thing.
Fertility is measurable in your own shop, and it holds still. The ranking of efficient against inefficient languages barely changes between formal and casual text, with a correlation above 0.97 across registers [4]. That makes it a per-language constant rather than a research project: push a sample of your real traffic through the tokenizer you actually call, divide tokens by words, and you have the multiplier for that language's line in the bill. The corpus skew that produced the vocabulary is upstream and not yours to fix [2]. Pricing it is.
The Polish comparison divides out to 1.45x [16], and it lands on three budgets at once: input tokens, output tokens, and the context window. Output is where the latency sits, because the model emits one token at a time, and thin vocabulary coverage makes that generation both slower and more expensive per unit of meaning [1].
The cost anecdote needs arithmetic before it needs sympathy. A 1.6x per-character penalty cannot triple a bill by itself. Even if the entire traffic mix had moved to those two languages, tokenization caps the increase at 1.6x, which leaves 1.875x to come from volume [17]. The tokenizer amplified growth without driving the tripling itself. Worth establishing before anyone goes hunting for a 3x saving in a vocabulary swap. The transferable figure is the fix: 35 percent off per-character token cost for those languages, with no change to the model or the prompt [8], which takes a 1.6x penalty to roughly 1.04x [18]. Near parity with English, from a tokenizer.
Quality is where the fertility story stops paying. The source offers a correlation, that higher compression ratio tracks lower task accuracy in a language [10]. The HRM8K benchmark tested the causal question on Korean against English and put the gap in comprehension of non-English input rather than in reasoning capability itself [11]. How well a model understands the input is not the same quantity as how many pieces the input was chopped into, and no vocabulary change addresses it. The Irish exam results make the distinction concrete: a 20.4 point spread [19], plus responses that stayed valid Irish less than 80 percent of the time [14]. A model drifting into English mid-answer is a generation-policy failure.
Two caveats limit what transfers here. The gap is reported as most pronounced on reasoning tasks dominated by high-resource languages and as not holding for factual questions or broader real-world queries [15], so a support-deflection workload should not budget for the exam-question spread. And the cost case comes to us as one breakdown of an unnamed team with no baseline traffic published [20], which is why the 35 percent is the number I would take into a review and the 3x is theirs.
What to watch
- Whether HRM8K's input-comprehension finding replicates on a language pair with a wider fertility spread than Korean-English.
- Whether any vendor publishes per-language fertility for its production tokenizer, so the token bill can be forecast before the traffic arrives.
- Whether the sub-80 percent valid-Irish response rate moves with tokenizer changes or only with generation-side constraints.