tiktoken keeps its only byte-pair trainer in a file named _educational.py

Encoding replays the merge list. Split the input into bytes, find the adjacent pair with the lowest merge rank, merge it, then repeat until nothing merges ^[11]. The educational encoder that does this is thirty-four lines and never iterates the merge list ^[11]. Training takes the most frequent pair and encoding takes the lowest-ranked one, which the post says comes to the same thing ^[12]. So the vocabulary is the merge list in order, and the post calls that order the whole model of the language ^[10].

Subwords exist because both ends of the range fail: characters give a tiny vocabulary and sequences far too long, while whole words give short sequences, an unbounded vocabulary and a hole for every word missing from it ^[4][5]. Sennrich, Haddow and Birch pointed byte-pair encoding, a compression algorithm from 1994, at that problem in 2016 ^[6]. The author read tiktoken's `_educational.py` and checked its encoder against the Rust core by running both over every sample in the post and comparing ids ^[8].

Order makes ties matter. The post's interactive widget runs the loop on a toy corpus, and twelve of its eighteen steps are ties, settled by the rule that the first pair seen wins ^[13]. Two thirds of that demonstration's merges were decided by iteration order ^[1]. Whether the ratio survives on a real corpus depends on how flat the pair counts are, and the post measures ties only on the toy corpus ^[13].

Because the merges run on bytes, any input encodes, and a character the merges never saw costs up to one token per UTF-8 byte instead of raising ^[14]. Under GPT-2 that puts the character ss-ligature `ß` at one token, `漢` at three, and a Gothic letter at four ^[15]. `o200k_base` has `漢` whole; the Gothic letter still costs four everywhere ^[16]. A lexer that met bytes it did not recognise would throw, and this one just bills you.

Attention costs the square of the sequence length ^[4]. Text made entirely of characters that encode at three tokens each occupies three times the positions of the same text under a vocabulary that holds them whole, so roughly nine times the attention work ^[2]. For that factor to transfer to your traffic, the documents have to be almost all such characters. Mixed text pays the penalty only on the share that never merged.

There is a second stage before any of this. A regex splits the text into chunks and merges never cross chunk boundaries, and GPT-2's pattern keeps a leading space attached to the word after it, so " token" and "token" are different tokens ^[17]. Every id has its own row in the embedding matrix ^[3]. Two strings differing only in a leading space therefore reach the first matrix multiply as different vectors ^[3].

That is why the vocabulary is fixed before training: the embedding rows were trained against those ids ^[3][1]. The author, who came to machine learning from compilers, wrote: "I came to this from the compiler side, where the parser is something you own and can change on a Tuesday. Here you own it exactly once, before training, and never again." ^[2]