ScienceNot yet confirmed elsewhere1 publisher2 min readPublished
Retrofit turns subword language models into byte-level ones with under 1% of a pretraining budget
Minixhofer and colleagues converted subword language models to read raw bytes using 49.1 billion training tokens, under 1% of a typical pretraining budget. Byte-level models have long trailed subword systems, and starting from a trained model is how this work closes most of that gap.
The Scientist · Science desk
What happened
- The authors report that the converted models excel on character-level reasoning tasks.
- Across various tasks, performance is close to that of state-of-the-art subword models and far above earlier byte-level models with similar parameter counts.
- All leading language models still rely exclusively on subword tokenization, even though many earlier byte-level papers claimed efficiency advantages.
- Nature's commentary notes that many token-based models say 'strawberry' has two 'r's, because they cannot see the characters inside a token.
Why it matters
- cost Whoever holds a trained subword model pays only for a conversion, so building a byte-level model no longer requires a pretraining budget in the trillions of tokens.
- decision Because tooling around the source model carries over, a team can run a byte-level variant beside the subword model it already serves and compare the two on its own data.
- constraint A byteified model starts from its subword parent, so byte-level quality under this method moves with whatever subword models are available to convert.
- precedent Byte-level wins measured against matched from-scratch baselines become a weak guide to production; the comparison future byte-level claims will need to beat is a converted strong model.
Earlier byte-level work chose a tidy control. Researchers trained a byte model from random initialization and compared it with a subword model also trained from scratch [14]. The authors argue that this picked the wrong opponent. Leading subword models carry accumulated work on data curation, architecture and post-training, and the authors say byte-level development cannot keep pace with that without extensive investment [15].
Byteification avoids training from scratch. It takes a trained subword model and converts it in two stages [3]. The authors give 49.1 billion tokens as under 1% of a typical pretraining budget, which puts that typical budget above 4.91 trillion tokens [16]. All of that is training cost. Inference is a separate bill. The authors call their speeds practical and attribute them to processing byte-level information efficiently [13].
The abstract's closing line says the work removes a long-standing performance barrier to end-to-end byte-level modelling [9]. The result beneath it is narrower: performance close to that of leading subword models [6]. The headline on Nature's commentary has a retrofitted model counting the letter 'i's in 'artificial intelligence' [12]. The commentary says byteified models keep the ability to read individual characters while performing competitively [11].
The authors say subword tokenization is a particular problem for code and biological sequences, because meaning there depends on individual characters or bytes [1]. The abstract and opening section do not report results for those domains, or throughput figures for inference. I think the cost result is the firm finding. Character access looks like a precondition for handling a variable name or a protein sequence byte by byte. Whether it improves code completion or sequence analysis has to be shown against the subword parent each converted model came from.
The paper lists two more subword problems that pushed researchers toward bytes: outputs biased by how a prompt happens to be tokenized, and vocabulary limits that in practice favour English [8].
What to watch
- Code and biological-sequence benchmarks that compare a byteified model with the subword model it was converted from.
- Measured inference throughput and latency for byteified models against their source models.
- Whether any leading model developer ships a byte-level model built by conversion.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence60
- Adoption5
- Hype gap+15
- Incentives
- Insufficient
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Subword tokenization obscures fine-grained information, which is problematic especially for scientific data such as computer code or biological sequences, where meaning depends on the individual characters or bytes.
- [2]
Models that operate directly on the byte encoding of text have until now lagged behind subword-based models in performance.
- [3]
The authors use a two-stage conversion procedure, which they call byteification, to retrofit existing subword-based models into byte-level models with minimal extra training.
- [4]
The authors retrofitted an existing subword-level LLM to the byte level using less than 1% of a typical pretraining budget, 49.1 billion tokens in total.
- [5]
The resulting byte-level models outperform earlier byte-level approaches and excel on character-level reasoning tasks.
- [6]
The byteified models achieve performance close to that of state-of-the-art subword-level LLMs across various tasks, while vastly surpassing earlier byte-level LLMs with a comparable parameter count.
- [7]
Many earlier byte-level LLMs claim to outperform subword-level LLMs on the efficiency-performance Pareto frontier, but byte-level LLMs have not seen widespread adoption and all leading LLMs still exclusively rely on subword tokenization.
- [8]
Subword LLMs are implicitly biased towards particular responses based on how the prompt is tokenized and are restricted in vocabulary size, which in practice leads to English-centricity; these problems motivated research into alternatives, most commonly UTF-8 bytes.
- [9]
The authors write that their results remove a long-standing performance barrier to end-to-end byte-level language modelling.
- [10]
Strawberry contains the letter 'r' three times, but many LLMs answer twice, because most LLMs encode words as tokens and cannot access the individual characters in each word.
- [11]
Writing in Nature, Minixhofer et al. report byteification, and show that byteified models can achieve competitive performance while retaining the ability to read individual characters.
- [12]
Nature's News & Views headline: a retrofitted LLM can count the letter 'i's in 'artificial intelligence'.
- [13]
The models achieve practical inference speeds by efficiently processing byte-level information, and adaptability by reusing the existing ecosystem around the source LLM.
- [14]
The authors hypothesize that byte-level models lagged in practice because existing approaches train a new byte-level model from random initialization and compare it against a subword-level LLM also trained from random initialization.
- [15]
Training of state-of-the-art subword LLMs is rapidly evolving, combining innovations in data curation, model architecture and post-training; keeping up with this pace is infeasible for byte-level LLM development without extensive investments, according to the authors.
- [16]
Because 49.1 billion tokens is under 1% of a typical pretraining budget, the implied typical budget exceeds 4.91 trillion tokens.
Sources
1 independent publisher whose own reporting we read for this story.
- nature.comRetrofitted LLM can count the letter ‘i’s in ‘artificial intelligence’
3 articles · October 6, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Byte-level language modelsFollow
- Large language modelsFollow
- Subword TokenizationFollow