Tokenizers are weirder than you think

When people first learn how large language models work, the tokenizer gets one slide: "text is split into tokens, roughly ¾ of a word each". Then everyone moves on to attention. That is a shame, because the tokenizer explains a whole class of strange behaviour that otherwise looks like the model being stupid.

Byte-pair encoding in one paragraph

Most modern tokenizers are variations of byte-pair encoding (BPE). You start with a vocabulary of single bytes. You scan a huge training corpus, find the most frequent adjacent pair, merge it into a new token, and repeat — tens of thousands of times. Frequent words end up as a single token, rare words are glued together from pieces, and anything at all can still be represented, because in the worst case you fall back to raw bytes.

The important consequence: the split is decided by frequency in the training data, not by linguistics. Tokens are not syllables or morphemes. They are whatever happened to be common.

Why the model can't count the r's

Ask a model how many times a letter appears in a word and it may get it wrong with total confidence. Look at what it actually receives:

"strawberry"  →  ["str", "aw", "berry"]   →  [496, 675, 15717]

(The exact split depends on the tokenizer; this is illustrative.) The model gets three opaque IDs. Nothing in ID 15717 says "this contains two r's" — the model has to have memorised the spelling of that token from training data. Usually it has, roughly. Sometimes it hasn't. Character-level tasks — counting letters, reversing strings, rhymes, anagrams — fight the representation instead of working with it.

Practical fix: if you need character-level reasoning, make the characters explicit. Ask the model to first spell the word with separators (s-t-r-a-w-b-e-r-r-y), which forces roughly one token per letter, then count. Or just do it in code.

The leading space is part of the token

In most BPE vocabularies, "hello" and " hello" (with a leading space) are different tokens. Words in the middle of a sentence almost always carry their space with them. This has a few fun consequences:

A prompt that ends with a trailing space can measurably degrade completions. The model now has to start a word without its usual leading-space token, which is a rarer path.
Capitalisation produces different tokens too: " Paris", "Paris" and " PARIS" may share nothing.
If you are doing classification by comparing log-probabilities of label tokens, make sure you compare the tokens the model would actually emit — usually the space-prefixed ones.

Numbers are chopped up inconsistently

Older tokenizers merged digits greedily, so 1234 might be one token while 1235 is two. Arithmetic then becomes a lookup problem over an irregular alphabet. Many newer tokenizers split numbers into single digits or fixed groups of up to three, which noticeably helps arithmetic — but you should not assume your model was trained that way. If precision matters, ask for a tool call, not mental arithmetic.

Your language has a price tag

Because merges are learned from frequency, languages that dominate the training corpus get efficient tokens. English text often averages around four characters per token. Text in languages with less representation — or in non-Latin scripts — can take two to three times as many tokens for the same meaning.

Effect	Why it matters
Higher cost	APIs bill per token, so the same request is simply more expensive.
Shorter effective context	A 32k-token window holds noticeably less text.
Slower generation	Output speed is tokens per second, not words per second.

It is worth measuring this for your actual data rather than trusting the ¾-word rule of thumb.

Measure, don't guess

Every serious model family ships its tokenizer. Counting tokens for a sample of your real inputs takes a few lines:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("your/model-name")
for text in samples:
    ids = tok.encode(text, add_special_tokens=False)
    print(len(text), len(ids), round(len(text) / len(ids), 2))

Two habits that pay for themselves: print the actual token strings (tok.convert_ids_to_tokens(ids)) whenever a model does something weird with a specific word, and keep a chars-per-token number for each language you serve. You will budget context windows and costs much more accurately.

Takeaway

The tokenizer is the one part of the stack you cannot fine-tune away. It is fixed at pre-training time, and everything downstream inherits its quirks. When a model fails at something a child could do — spelling, counting, simple sums — check the tokens before you blame the weights.