Vocabulary
Understand the tokenizer vocabulary as the fixed inventory of text units a model can process — and why vocabulary size is one of the most important engineering trade-offs in tokenizer design.
Vocabulary: The Total Boundaries of the Scholar's World
The Vocabulary is the master dictionary book bound before the scholar begins training. If $|V| = 128,000$ (like GPT-4o or LLaMA 3), the scholar has exactly 128,000 distinct concept blocks. A larger dictionary allows speaking in longer, richer words (lower token count), but requires a thicker memory table (larger $[V \times D]$ embedding parameter footprint).
IntroductionWhat is the vocabulary?
The vocabulary is the complete, finite set of tokens the tokenizer knows. Every token in the vocabulary has exactly one unique integer ID. The model can only process tokens that live inside its vocabulary — everything else must be decomposed into subword pieces that do exist, or, in the worst case, mapped to a special [UNK] (unknown) token.
What vocabulary is
- A fixed mapping: token string → integer ID
- Built once from a training corpus, never changed at inference time
- Contains whole words, subword pieces, special tokens, and characters
- Stored as a
vocab.jsonfile alongside the tokenizer
What vocabulary enables
- Converts raw text into a sequence of integer IDs the model can process
- Provides a shared language between tokenizer and model embedding layer
- Guarantees every token maps to exactly one row in the embedding matrix
- Allows the model to reconstruct text from IDs (decoding)
- The vocabulary is a dictionary that is locked when training ends.
- The model's entire universe of text is bounded by what's in this dictionary.
- You cannot add or remove entries without modifying the embedding layer and retraining.
WhyThe vocabulary size trade-off
Vocabulary size is one of the most consequential hyperparameters in tokenizer design. It cannot be changed after training — it determines the shape of the entire model's embedding layer and affects every downstream inference cost.
⬇️ Too-small vocabulary
- Many common words split into multiple pieces → sequences get longer
- Longer sequences are slower and more expensive to process
- Semantic meaning fragmented across multiple tokens
- Rare words, proper nouns, and domain terms suffer most
⬆️ Too-large vocabulary
- Embedding matrix is huge → more GPU memory, slower training
- Many entries appear rarely → poor embedding quality
- Rare tokens get few gradient updates during training
- Larger model checkpoint files, slower loading
Real vocabulary sizes chosen by major models:
| Model | Vocab Size | Algorithm | Notes |
|---|---|---|---|
| BERT-base | 30,522 | WordPiece | English, cased and uncased variants |
| GPT-2 | 50,257 | BPE (byte-level) | Byte-level — zero UNK tokens possible |
| LLaMA 2 | 32,000 | SentencePiece BPE | Multilingual coverage |
| GPT-4 (est.) | ~100,000 | BPE (tiktoken) | Larger vocab reduces fragmentation |
| T5 | 32,100 | SentencePiece | Shared encoder/decoder vocab |
IntuitionThe Scrabble tile bag analogy
Imagine the vocabulary as a pre-printed set of Scrabble tiles in a bag. You can spell any word, but only using the tiles that already exist in the bag. You cannot manufacture a new tile mid-game — that would break the entire scoring system. Adding a new tile requires reprinting the full set from scratch.
- If the tile "AI" is in the bag as one tile → great, one token.
- If only "A" and "I" are in the bag → you need two tiles:
A+I. - If subword pieces exist → "tokenization" might use tiles
token+##ization. - If truly nothing matches → the special [UNK] tile is played (information is lost).
- The vocabulary shapes how meaning is represented.
- A larger vocabulary can express more concepts as atomic units, reducing fragmentation.
- Smaller vocabularies trade coverage for a smaller embedding table.
TheoryWhat lives inside a vocabulary
A tokenizer vocabulary contains four types of entries, each serving a distinct role:
a) Whole common words
High-frequency words that appear often enough to earn their own entry. Examples: the, is, play, model, data.
b) Subword pieces
Word fragments with special markers. BERT uses ## prefix for continuations: ##ing, ##tion, ##ness. GPT models use no marker (BPE merges handle it implicitly).
c) Special tokens
Functional tokens with no text meaning: [PAD] (padding), [UNK] (unknown), [CLS] (classification), [SEP] (separator), <s>, </s>.
d) Individual characters (fallback)
Single letters, digits, punctuation. Added first so the tokenizer can always decompose any word, even novel ones. a–z, A–Z, 0–9, ,, ., etc.
The vocabulary is built by running BPE or WordPiece on a training corpus until the target vocab_size is reached. Each entry maps to exactly one row in the model's embedding matrix.
## prefix. In GPT/BPE, the space before a word is encoded as part of the first token using a special space character (Ġ in GPT-2). Always check the tokenizer documentation for its specific convention.How It WorksVocabulary training and vocab.json
Building a vocabulary is a training process performed once, before any model training begins. The result is a vocab.json file that the tokenizer loads at inference time.
- Start: Add all individual characters (and special tokens) — these always survive.
- Iterate: Repeatedly merge the most frequent adjacent pair into a new entry, adding it to the vocabulary.
- Stop: When total vocabulary entries reach the target size (e.g., 30,000 or 50,000).
- Save: Write the final
token → integer_idmapping tovocab.json.
// Simplified vocab.json (BERT-style)
{
"[PAD]": 0,
"[UNK]": 100,
"[CLS]": 101,
"[SEP]": 102,
"[MASK]": 103,
"!": 999,
"the": 1996,
"model": 2944,
"token": 19204,
"##ing": 2075,
"##tion": 3508,
"##ness": 4757,
"un": 4895,
"##believ":26354,
"##able": 3085,
"z": 1062
// ... 30,522 total entries
}CalculateEmbedding table size
Each vocabulary entry corresponds to one row in the model's embedding matrix. You can calculate the exact memory cost of the embedding layer using this formula:
← just the embedding layer of GPT-3!
MathematicsEmbedding matrix and parameter scaling
The vocabulary ties directly to two core mathematical relationships in the model:
Embedding matrix shape
Shape: [V, D] where V = vocab_size, D = embedding_dim.
Each token ID i selects row i from this matrix. The lookup is a simple index operation (not a matrix multiply).
Doubling V doubles the embedding layer's parameter count — directly impacting model size, memory, and training compute.
Vocabulary coverage rate
A vocabulary covers a corpus if its tokens can represent any text without resorting to UNK.
coverage = |tokens_in_vocab ∩ corpus_tokens| / |unique_corpus_tokens|
For byte-level BPE (GPT-2), coverage = 1.0 always. For WordPiece, coverage ≈ 0.999+ on English text.
$V=30k,\; D=768 \;\rightarrow\; 23.0\text{M params}$
$V=60k,\; D=768 \;\rightarrow\; 46.1\text{M params}\;(2\times)$
$V=120k,\; D=768 \;\rightarrow\; 92.2\text{M params}\;(4\times)$
[D, V] — predicting the next token requires scoring all V vocabulary entries. This means large vocabulary also costs in the output layer, often weight-sharing with the embedding matrix to save memory.VisualizeVocabulary table & live search demo
The table below shows 10 entries from a representative vocabulary — covering all four entry types. IDs are illustrative BERT-style values.
| ID | Token | Type | Notes |
|---|---|---|---|
| 0 | [PAD] | Special | Padding — fills empty sequence positions |
| 100 | [UNK] | Special | Unknown token — last resort for missing text |
| 101 | [CLS] | Special | Classification / start-of-sequence marker |
| 102 | [SEP] | Special | Separator between two text segments |
| 1996 | the | Whole Word | Most common English word — earns its own entry |
| 2944 | model | Whole Word | Common NLP term — one token |
| 19204 | token | Whole Word | Common enough to be in BERT's vocab |
| 2075 | ##ing | Subword Piece | Suffix — attaches to word stems |
| 3508 | ##tion | Subword Piece | Suffix — "tokenization", "attention" |
| 1037 | a | Character | Single character — ultimate fallback |
Interactive Vocabulary Search — type a word or phrase to see how each word maps against a toy 30-word vocabulary. Green = in vocab as whole word, amber = decomposed into pieces, red = no valid decomposition ([UNK]).
CodeInspecting a real vocabulary in Python
from transformers import AutoTokenizer
# Load a real tokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
# ── 1. Basic vocabulary stats ──────────────────────────────
print("Vocab size:", tok.vocab_size) # 30522
print("Model max input:", tok.model_max_length) # 512
# ── 2. Full vocab dict: token → id ────────────────────────
vocab = tok.get_vocab()
print("Type:", type(vocab)) # <class 'dict'>
print("Sample entries:", dict(list(vocab.items())[:5]))
# ── 3. Find all subword pieces ─────────────────────────────
subwords = {k: v for k, v in vocab.items() if k.startswith("##")}
print(f"Number of ## subword pieces: {len(subwords)}") # ~10k
top10 = sorted(subwords.items(), key=lambda x: x[1])[:10]
print("Top subword pieces:", top10)
# ── 4. Find all special tokens ─────────────────────────────
print("Special tokens map:", tok.special_tokens_map)
# {'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]',
# 'cls_token': '[CLS]', 'mask_token': '[MASK]'}
print("All special token IDs:", tok.all_special_ids)
# ── 5. Token ↔ ID lookups ──────────────────────────────────
token_id = tok.convert_tokens_to_ids("token")
print(f"ID of 'token': {token_id}") # 19204
token_str = tok.convert_ids_to_tokens(19204)
print(f"Token at ID 19204: '{token_str}'") # 'token'
# ── 6. Check if word is a single token ─────────────────────
word = "tokenization"
ids = tok.encode(word, add_special_tokens=False)
pieces = tok.convert_ids_to_tokens(ids)
print(f"'{word}' → {pieces}")
# ['token', '##ization'] ← 2 pieces, not in vocab as whole word
# ── 7. Scan for domain subword fragmentation ───────────────
domain_words = ["transformer", "attention", "embedding", "gradient", "tokenization"]
for w in domain_words:
ids = tok.encode(w, add_special_tokens=False)
pieces = tok.convert_ids_to_tokens(ids)
flag = "✓ single" if len(ids) == 1 else f"✗ {len(ids)} pieces: {pieces}"
print(f" {w:20s} → {flag}")len(tok.encode(word, add_special_tokens=False)) == 1 tells you whether a word is a single token. Words with more than 1 ID are split into subword pieces.Hands-onVocabulary exploration exercises
Complete all five exercises to build intuition you can't get from reading alone:
Expected output — Exercise 5 verification:
CompareSmall vs Medium vs Large vocabulary
| Attribute | Small (30k) | Medium (50k) | Large (100k) |
|---|---|---|---|
| Example model | BERT-base, LLaMA 2 | GPT-2 | GPT-4 (est.), Falcon-180B |
| Avg sequence length | Longer (more splits) | Moderate | Shorter (fewer splits) |
| Embedding table (D=768) | ~93 MB | ~154 MB | ~308 MB |
| Rare word coverage | Poor — many splits | Moderate | Good — most rare words whole |
| Multilingual quality | Poor unless designed for it | Moderate with byte-level | Good if trained on diverse corpus |
| API cost (tokens/word) | Higher (more tokens/word) | Moderate | Lower (fewer tokens/word) |
ApplyVocabulary decisions in real-world scenarios
🔍 Building RAG systems
Vocabulary coverage affects how domain-specific text tokenizes. Medical terms like "thrombocytopenia" or legal terms like "indemnification" may fragment heavily in a general-purpose vocabulary, spreading meaning across many tokens and reducing retrieval quality.
🎯 Fine-tuning on domain text
If your target domain has specialized terms (scientific, legal, code), audit vocabulary coverage first. High fragmentation of domain terms leads to worse downstream performance. Consider tokenizers trained on similar domains (BioBERT, CodeBERT).
➕ Expanding vocabulary with add_tokens()
Call tok.add_tokens(["[NEW_ENTITY]"]) and then model.resize_token_embeddings(len(tok)). This adds new rows to the embedding matrix — but those rows are randomly initialized. You must fine-tune with examples of the new token for the model to learn its meaning.
💰 API cost implications
With OpenAI / Anthropic APIs you pay per token. Domain-specific terms that fragment into many tokens cost more. "GPT-4" → 1 token. "Acetylcholinesterase" → 6+ tokens. Knowing tokenizer coverage of domain terms helps estimate API costs accurately.
LimitationsWhat vocabulary can't do and common pitfalls
❌ Common mistakes
- Assuming you can swap tokenizers without retraining — every model is tied to its specific vocabulary
- Thinking larger vocabulary is always better — it costs more in memory and training
- Ignoring vocabulary when fine-tuning domain models — poor coverage degrades performance
- Counting words instead of tokens when estimating API costs or context window usage
✅ Better mental models
- Vocabulary is model-specific — always check which tokenizer a model uses
- Coverage matters more than raw size — a smaller well-targeted vocabulary can outperform a larger general one
- Vocabulary encodes biases from training corpus — over/under-representation of languages affects tokenization efficiency
- Always measure token counts (not word counts) when working with LLM APIs
the in BERT but a completely different token in GPT-2. Never mix token IDs across models.Feynman ChallengeExplain it like a pre-printed Scrabble tile set
Core analogy: A tokenizer's vocabulary is like a pre-printed Scrabble tile set. Before the game starts, the manufacturer decides exactly which tiles to include and how many of each. Once the box is printed and shipped, no new tiles can be added mid-game. If you need to spell a word whose letters aren't in the set, you must use combinations of what you have — or forfeit the word entirely.
The model's vocabulary is the box. The training run is the printing process. Inference is the game. You cannot add tiles during the game.
Can you explain — without notes:
- Why the vocabulary is fixed at training time
- What happens when a word isn't in the vocabulary
- Why changing the vocabulary requires retraining the embedding layer
- The trade-off between small and large vocabularies in concrete terms
🚫 Common Misconceptions
❌ "Larger vocab = smarter model"
False. A larger vocabulary only changes the length of inputs the model processes. Reasoning and intelligence are defined by the depth, weights, and training of attention layers — not vocabulary size.
❌ "Every token has a well-trained embedding"
False. Due to Zipf's Law, rare tokens (specialized coding terms, niche languages) appear rarely during pre-training. Their embedding vectors often remain undertrained, which is why models sometimes produce poor output for rare tokens.
❌ "You can swap tokenizers without retraining"
False. Each model is architecturally coupled to its specific vocabulary. The embedding matrix has exactly V rows — one per vocabulary entry. Using a different tokenizer would misalign every token ID.
❌ "Token IDs mean the same thing across models"
False. Token ID 1996 means the in BERT and something entirely different in GPT-2. IDs are model-specific and must never be mixed across tokenizers.
BuildMini-projects to reinforce mastery
🔨 Project Ideas
🔭 Vocabulary Inspector CLI
Given a model name and a word, print the token pieces, their IDs, their types (special/whole/subword/character), and whether the word is in-vocab as a single entry.
📊 Coverage Analyzer
Given a text file and a tokenizer, compute: (1) total unique words, (2) percentage that are single tokens, (3) average token count per word, (4) top-20 most-fragmented words.
🧮 Embedding Size Calculator
Web form: enter vocab_size, embedding_dim, and data type (fp32/fp16/bf16). Compute total parameters, memory in MB/GB, and show how memory scales if vocab doubles.
⚖️ Vocabulary Comparator
Load two tokenizers (BERT vs GPT-2). For a given sentence: show side-by-side tokenization, token counts, IDs, and piece breakdown. Highlight tokens that appear in both vocabularies.