After this lesson, you will be able to: Explain why text must be turned into a fixed vocabulary of tokens, the tradeoffs of character / word / sub-word granularity, and what out-of-vocabulary really means.
Every model needs a finite input alphabet, but language is open-ended. This lesson frames the tokenization problem, the core design decision under every LLM, before the next lesson trains a real tokenizer.
A model's input is a sequence of integers, each an index into a fixed vocabulary. Language has effectively unlimited words (new names, typos, code, other languages), so you cannot just map every word to an integer. Pick word-level and you get a huge vocabulary and an out-of-vocabulary (OOV) problem the moment you see a word you never trained on. Pick character-level and there is no OOV, but sequences become very long and the model must learn spelling before meaning. Sub-word tokenization is the compromise the whole field settled on.
Sub-word units keep common words whole and split rare ones into reusable pieces.
# Conceptually, a sub-word tokenizer learns a vocabulary of frequent pieces:# 'tokenization' -> ['token', 'ization']# 'antidisestablishmentarianism' -> ['anti', 'dis', 'establish', 'ment', ...]# 'GPT-4' -> ['G', 'PT', '-', '4']## Benefits:# * No OOV: any string decomposes into known pieces (down to bytes).# * Bounded vocab (e.g. 32k-128k) keeps the embedding table a sane size.# * Common words stay single tokens, so sequences stay short.## This is why 'Hello world' might be 2-3 tokens but a rare technical term# is many, and why token count (not word count) drives API cost + context use.
Tokenization quietly shapes everything downstream. It is why models miscount the letters in 'strawberry' (they see tokens, not characters). It penalizes languages whose scripts get split into many tokens, a real fairness and cost issue in multilingual NLP. It affects arithmetic (numbers tokenize inconsistently) and code. Papers have been written on each of these. When you read 'we use the Llama tokenizer,' that is a real experimental choice with consequences.
Comparing two models on 'tokens per second' without noting they use different tokenizers (so a token means different amounts of text). Reporting sequence lengths in words when the model budget is in tokens. Assuming whitespace splitting is 'tokenization' (it is not, for any modern model). Forgetting that special tokens (BOS, EOS, padding, chat-template markers) occupy vocabulary slots and context budget. Ignoring that the same text tokenizes differently across model families, which breaks naive cross-model comparisons.
Pick the precise reason.
Sign in and purchase access to unlock this lesson.