project
OpenAI's open-source byte-pair encoding tokenizer library, used to count and split text into tokens via encodings like cl100k_base and o200k_base.
No evidence-backed relationships are recorded.
Habr user donseo's test of 11 tokenizers found Claude Opus 5 turns Russian into 2.96 times the tokens of the same English text. Anthropic bills $5 per million input tokens in either language, so Russian on Opus 5 costs nearly three times as much to send.
Reality
Encoding replays an ordered merge list that was learned once, before training. Swap the list and the weights still load, but the model you tested is gone.
Reality
A dev.to writeup of the LectuLibre book translation pipeline sets chunks at 3,000 tokens with 500 of overlap and keeps a glossary in Postgres. An output ceiling sets the shape of the job, and it holds however big the input window gets.
Reality
Decodo's own runs on ten pages put raw HTML at 3 to 24 times the token count of the same pages as Markdown, with every model still answering correctly and the JSON-LD fields lost in the conversion.
Reality
LectuLibre's dev.to write-up splits an EPUB on its own chapters and paragraphs, caps each chunk near 4,000 tokens, and feeds a glossary of proper nouns extracted from earlier chunks back into every translation prompt.
Reality
One measurement puts a 50-server tool catalog at 56 percent of a 128K context window before the first prompt. The per-server arithmetic is the part worth budgeting.
Reality
A zero-budget git-to-Markdown pipeline works because a deterministic fallback runs when the free model tier says no. The interesting part is what the degraded output costs.
Reality
A developer measured 111,713 tokens of JSON schema from 10 MCP servers, injected before the first message, and priced the year at $1,764 per seat. The per-server number is the useful one.
Reality
A dev.to benchmark says one connector carries 42 percent of the schema load while supplying 3.7 percent of the tools. The direction holds. The money math does not.
Reality
A 452-configuration benchmark of the parmar pre-filter splits two effects that looked like one: denser tokens buy a flat 15% on gzip, while the gain that scales comes from window expansion.
Reality
A standing measurement of 14 MCP servers finds Claude's tokenizer counts schema text a median 64.1 percent above tiktoken, the counter every published cost study uses.
Reality
Two models can quote identical per-token prices and still bill differently for the same string. The split is decided by merge tables you did not train and cannot assume.
Reality