Tokens

A token is the smallest discrete unit of text that a language model processes. It is not a word, not a character, not a sentence — it is a model-specific piece chosen by a tokenizer algorithm trained on text data.

Each subtopic below is a fully elaborated learning page with interactive demos, code, Feynman challenges, and mini-projects. Study them in the order shown for best results.

Every tokenizer — regardless of model — transforms raw text through these five stages before handing integers to the neural network.

flowchart LR
    A["📝 Raw Text\n'Hello World!'"] --> B["🔧 Normalize\nlowercase, unicode"]
    B --> C["✂️ Pre-tokenize\nsplit on spaces/punct"]
    C --> D["🔬 Subword model\nBPE / WordPiece / SP"]
    D --> E["🏷️ Post-process\nadd [CLS] [SEP]"]
    E --> F["🔢 Encode\ntoken→integer ID"]
    F --> G["📦 Model Input\n[101, 7592, 2088, 102]"]
    style A fill:#1a1e2e,stroke:#252a40,color:#b0b8d8
    style G fill:#1a1e2e,stroke:#3b82f6,color:#93c5fd
        

These are the operations you'll use in every LLM project. Run them, inspect them, understand every line before moving to embeddings.

from transformers import AutoTokenizer

# ── Load a tokenizer ──────────────────────────────────
tok = AutoTokenizer.from_pretrained("bert-base-uncased")

# ── Encode text → token IDs ───────────────────────────
text = "The tokenizer is surprisingly useful."
ids  = tok.encode(text)
print("IDs:", ids)
# → [101, 1996, 19204, 17629, 2003, 14477, 6937, 1012, 102]

# ── See the token strings ─────────────────────────────
tokens = tok.tokenize(text)
print("Tokens:", tokens)
# → ['the', 'token', '##izer', 'is', 'surprisingly', 'useful', '.']
# Note: 'tokenizer' → ['token', '##izer']  (two tokens, one word!)

# ── Decode IDs back to text ───────────────────────────
decoded = tok.decode(ids, skip_special_tokens=True)
print("Decoded:", decoded)

# ── Vocabulary size ───────────────────────────────────
print("Vocab size:", tok.vocab_size)   # → 30522

# ── Special token IDs ─────────────────────────────────
print(tok.special_tokens_map)
# → {'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]',
#    'cls_token': '[CLS]', 'mask_token': '[MASK]'}

# ── Token count vs word count ─────────────────────────
words  = len(text.split())
n_toks = len(tok.tokenize(text))
print(f"Words: {words}, Tokens: {n_toks}, Ratio: {n_toks/words:.2f}x")
# → Words: 5, Tokens: 7, Ratio: 1.40x

⚠️ The 5 Most Common Misconceptions

• Definition: A token is a model-defined text piece — the atomic unit a language model processes.
• Not words, not chars: Tokens are subword units chosen by an algorithm trained on corpus statistics.
• Token IDs: Integer row indices into the embedding matrix. No semantic ordering.
• Vocabulary: Fixed set of all known tokens. Built once, cannot change without retraining.
• Subwords solve two problems: word-level fails on rare words; character-level produces too-long sequences.
• BPE (MUST): Iteratively merges most-frequent adjacent pair. Used by GPT-2, GPT-4, LLaMA.
• WordPiece (GOOD): Likelihood-based selection + MaxMatch greedy segmentation. Used by BERT.
• SentencePiece (GOOD): Raw-text, no pre-tokenization needed. Language-agnostic. ▁ marker.
• Pipeline (MUST): 5 stages: normalize → pre-tokenize → subword → post-process → encode. Model-specific.
• Special tokens: [CLS], [SEP], [PAD], [MASK], <|endoftext|> — structural, not content.
• Context windows: Measured in tokens, not words. ~1.3 tokens/word for English. 3–8x for non-Latin scripts.
• Next: Token IDs → Embedding lookup → Vectors → Attention → Transformer.