Brandeis's CPU profile puts 88% of a warm bert-base-uncased encoding pass in normalization
Simon Brandeis reports that text normalization took 88% of a warm English encoding pass with bert-base-uncased, against 8% for WordPiece. His faster normalizer was still a pull request at publication, so the token-parity check falls to whoever adopts it.
The Engineer · Build desk
Key figures: BERT encoding up to 5.7 times faster for the proposed branch against main; about 50 times faster in total against tokenizers 0.23; ModernBERT's English result improves 1.06 times, since its NFC path already skips ASCII on main.
| Figure | Value | Note | Claim |
|---|---|---|---|
| BERT encoding, proposed branch vs main | 5.7 times, up to | 13 | |
| Total vs tokenizers 0.23 | 50 times, about | 13 | |
| ModernBERT English gain | 1.06 times | 18 |
What happened
- The accent-removal step checked for nonspacing marks by searching a sorted table of 1,567 characters, and the pipeline ran Unicode property lookups even on ASCII letters.
- The proposed branch inspects 16 bytes at once with vector instructions, since ASCII bytes have a clear high bit, and so finds runs that need no Unicode normalization.
- For other scripts, a page cache fills answers for 256 code points on first encounter, so later characters from that page cost a memory read instead of a binary search.
- Brandeis reports up to 5.7 times faster BERT encoding for the branch against main, and about 50 times in total against tokenizers 0.23 once earlier improvements are counted.
- Output token hashes matched across the tested corpora and implementations, according to the write-up of his reproduction appendix.
Why it matters
- Who has to decide decision A team with a slow embedding or RAG service has to separate text preparation from model execution in its profile, since a faster GPU can leave a CPU-side bottleneck intact.
- What it limits constraint A service that handles mostly Chinese text or very short inputs cannot borrow the 5.7x figure as its own, because Chinese barely benefits in the BERT benchmarks and some tiny inputs got slower.
- Who's exposed exposure Matching hashes support output consistency only for the tested inputs, so a team with different text has to compare token IDs on its own corpus before swapping in a faster normalizer.
In DEV Community's write-up of Brandeis's October 15 Hugging Face article [3], normalization took 11 times as long as WordPiece [25]. The two together cover 96% of the pass. That leaves 4% for everything else [26]. The write-up's gloss is that a tokenizer sounds like a boundary-finding problem, but this profile mostly measured the cost of preparing characters [24].
BertNormalizer runs before WordPiece. It cleans text, inserts spaces around Chinese characters, decomposes Unicode, strips accents and lowercases the result [4]. Those steps help determine which token IDs the model receives, so a change to the implementation needs a check beyond throughput [5].
Take the letter s. A general-purpose Unicode path may decode the character, query its properties and run it through normalization. Its UTF-8 byte already shows it is ASCII [7]. In the branch, ASCII runs skip Unicode normalization, and lowercasing and whitespace handling still apply, also done with vector operations [9]. We think the design is sound: it first establishes that text can be left alone, then skips a transformation that would do nothing [12]. Unchanged stretches avoid copying, and an unchanged input returns without an allocation [11].
The runs were single-threaded, on two AMD EPYC hosts and an Intel Xeon host, with implementations alternated across rounds [14]. The shortcut helped English and source code most in the reported corpora [16]. ModernBERT shows how much depends on the tokenizer: on main, its NFC path bypasses ASCII already, and its English speedup comes to only 1.06 times [18].
For the 88% to describe another service, that service needs BertNormalizer in front of WordPiece and text close to those corpora [1][16]. It also needs tokenization on the CPU path of the request. The write-up says these are tokenizer results and do not measure a complete RAG request, GPU inference or application latency [19]. It adds that the profile does not establish that tokenization dominates any particular service [20].
We'd treat the 88% as a reason to run your own profile. The write-up proposes a comparison that holds the tokenizer configuration and input corpus fixed and measures both encoding time and complete request latency [23]. It would also compare token IDs [23].
What to watch
- Whether the normalizer pull request merges into tokenizers main, and in which release.
- A reproduction on Chinese and very short inputs, where the reported gains were smallest or negative.
- End-to-end RAG request timings with normalization split out; the reported benchmarks stop at the tokenizer.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence62
- Adoption5
- Hype gap−5
- Incentives
- Not scored
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Normalization consumed 88% of a warm English encoding pass with bert-base-uncased.
- [2]
WordPiece, the algorithm that chooses tokens, took 8% of the same warm encoding pass.
- [3]
Simon Brandeis reports the measurements in his October 15 Hugging Face article, which follows the work from a CPU profile to a proposed implementation.
- [4]
BertNormalizer runs before WordPiece. It cleans text, inserts spaces around Chinese characters, decomposes Unicode characters, strips accents and lowercases the result.
- [5]
The normalizer steps help determine which token IDs the model receives, so changing the implementation needs a check beyond throughput: the tokens must still match.
- [6]
Brandeis found a costly check for nonspacing marks, used when the normalizer removes accents, that searched a sorted table of 1,567 characters; the pipeline did Unicode property lookups even for ASCII letters that could not need decomposition or accent removal.
- [7]
A general Unicode path can decode the letter s, look up its properties and pass it through normalization, but its UTF-8 byte already shows it is ASCII.
- [8]
The proposed branch uses vector instructions to inspect 16 bytes at once; ASCII bytes have a clear high bit, so checking those bits lets the normalizer find a run that does not need Unicode normalization instead of decoding each character.
- [9]
BERT's other ASCII transformations, including lowercasing and whitespace handling, still apply in the proposed branch, which handles them with vector operations too.
- [10]
For other scripts, a page cache stores whether normalization could change a character or combine it with a neighbor; the first encounter fills answers for 256 code points, and later characters from that page use a memory read instead of another binary search. Russian, Greek and Arabic gain despite containing much less ASCII.
- [11]
Unchanged stretches avoid copying, and an unchanged input can return without an allocation.
- [12]
The implementation first establishes that it can leave text alone, then avoids paying for a transformation that would do nothing.
- [13]
Brandeis reports up to another 5.7 times faster BERT encoding for the proposed branch, bringing the total to about 50 times over tokenizers 0.23. The extra gain compares the branch with main; the larger headline includes earlier improvements over the older release.
- [14]
The reproduction appendix describes single-threaded runs on two AMD EPYC hosts and an Intel Xeon host, alternating the implementations across rounds.
- [15]
The reproduction compares output token hashes, and those hashes match across the tested corpora and implementations. Matching hashes support output consistency for those tests, leaving other inputs to be checked.
- [16]
The ASCII shortcut helps English and source code most in the reported corpora.
- [17]
Chinese barely benefits in the BERT benchmarks, and some tiny inputs become slower.
- [18]
ModernBERT's NFC path already skips ASCII on main, so its English result improves only 1.06 times.
- [19]
These are tokenizer results. They do not measure a complete RAG request, GPU inference or application latency.
- [20]
The profile does not establish that tokenization dominates any particular service.
- [21]
Brandeis says the changes were still in a pull request when he published.
- [22]
An application profile needs to distinguish text preparation from model execution; otherwise a faster GPU can leave the original CPU bottleneck intact.
- [23]
A useful comparison would keep the tokenizer configuration and input corpus fixed, then measure both encoding time and complete request latency. It would also compare token IDs.
- [24]
A tokenizer sounds like a problem about finding token boundaries, yet this profile mostly measured the cost of preparing characters.
- [25]
Normalization took 11 times as long as WordPiece in the profile.
- [26]
Normalization and WordPiece together take 96% of the pass, leaving 4% for everything else.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toBERT's tokenizer bottleneck was hiding before WordPiece
1 article · October 15, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- SIMD optimizationFollow
- Inference performanceFollow
- Subword TokenizationFollow