Training a Tokenizer: BPE, WordPiece, SentencePiece

After this lesson, you will be able to: Train a sub-word tokenizer yourself and explain how Byte-Pair Encoding, WordPiece, and SentencePiece/Unigram differ.

This is the first hands-on research skill: building the vocabulary a model will use. You will train a Byte-Pair Encoding tokenizer on a small corpus with the Hugging Face tokenizers library and understand the two other algorithms you will meet in papers.

Prerequisites:Text as Data: Tokens and Vocabulary

Byte-Pair Encoding (BPE), the dominant algorithm

BPE starts from a base vocabulary (bytes or characters) and repeatedly merges the most frequent adjacent pair into a new token, recording each merge. After N merges you have a vocabulary of base units plus N learned pieces, and the ordered merge list is how you tokenize new text. GPT-2/3/4 and Llama use byte-level BPE. The training is a frequency count plus a merge loop; the elegance is that the same merge rules apply to text never seen before.

Train a BPE tokenizer with the Hugging Face tokenizers library

A complete, runnable training loop on any text corpus.

python

from tokenizers import Tokenizer, models, trainers, pre_tokenizers

# 1. Start from an empty BPE model
tok = Tokenizer(models.BPE(unk_token="[UNK]"))
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)

# 2. Configure training: target vocab size + special tokens
trainer = trainers.BpeTrainer(
    vocab_size=8000,
    special_tokens=["[UNK]", "[PAD]", "[BOS]", "[EOS]"],
)

# 3. Train on your corpus (a list of file paths, or an iterator of strings)
tok.train(["corpus.txt"], trainer)

# 4. Use it
enc = tok.encode("Tokenization is a research choice.")
print(enc.tokens)   # the sub-word pieces
print(enc.ids)      # the integer ids the model actually sees
tok.save("my-tokenizer.json")

WordPiece and SentencePiece/Unigram

WordPiece (BERT) is BPE's cousin: instead of merging the most frequent pair, it merges the pair that most increases the likelihood of the training corpus under a unigram language model. SentencePiece is a library (not an algorithm) that treats the input as a raw stream including whitespace, so it is language-agnostic and reversible; it usually runs the Unigram algorithm, which starts with a large vocabulary and prunes pieces to maximize corpus likelihood. In papers: GPT/Llama use byte-level BPE, BERT uses WordPiece, T5/many multilingual models use SentencePiece Unigram. Knowing which is which lets you read a methods section correctly.

Common mistakes only experienced researchers catch

Training a tokenizer on the test text (leakage that inflates results). Setting vocab size by guess instead of measuring sequence-length and OOV-rate tradeoffs. Forgetting to add the special tokens the model architecture needs (BOS/EOS/PAD, chat-template tokens). Mixing a tokenizer trained on one corpus with a model pretrained on another. Comparing models across different tokenizers without acknowledging that 'token' is no longer a fixed unit.

Quick Check

A paper says it uses byte-level BPE. What does 'byte-level' guarantee?

Pick the correct guarantee.

←Text as Data: Tokens and Vocabulary

Back to Transformers and NLP Foundations

Embeddings and Positional Encoding→