Words vs Tokens

When you write text, you think in words. When a language model reads text, it operates on tokens. These are different units, and conflating them is one of the most common mistakes when working with LLMs.

In the infancy of NLP, the standard approach was whitespace-splitting (word tokenization): split sentences on spaces and punctuation to build a dictionary of distinct words. As models scaled, this simple rule ran into devastating mathematical bottlenecks.

The Word-Level Dictionary Bottleneck

• Infinite Vocabularies: New words ("rizz", "cryptocurrency"), jargon, and acronyms appear constantly. A word-level dictionary can't adapt without full re-training.
• Morphological Blindness: "run", "running", "ran", and "runner" share a semantic root. In a word-level system they are completely independent indices with no shared statistical history — extremely inefficient.
• Foreign Scripts: Languages without spaces (Chinese, Japanese, Thai) are entirely unhandled by whitespace splits.

Subword tokens resolve this: By breaking words into high-frequency fragments we capture roots and affixes ("##ing", "##er"), allowing neural weights to share meaning across hundreds of related word forms.

Word count ≠ token count. This is not a minor technicality — it directly affects your costs and capability limits.

• Pricing: OpenAI, Anthropic, and Google APIs charge per token. Estimating cost in words means you'll consistently underestimate.
• Context limits: A 128k-token context window is measured in tokens. A document with 100,000 words may be 130,000–150,000 tokens — already over the limit.
• RAG chunk sizes: Chunking documents for retrieval must respect token limits, not word limits. A "200-word chunk" is roughly 260–300 tokens.
• Prompt budgets: Planning a prompt by word count and being surprised when it exceeds the context window is a very common engineering mistake.

Think of words as book titles and tokens as pages. A short, familiar book title (a common word) occupies just one page. A long compound title (a rare or complex word) might need many pages to fully explain. The model only ever reads pages — it never sees titles.

A simple, frequent word like "the" stays as one token. A long or rare word like "unbelievable" gets split into smaller reusable pieces:

Word:   "unbelievable"          (1 word, 12 chars)
Tokens: ["un", "##believ", "##able"]   (3 tokens — BERT-style)

Word:   "tokenization"          (1 word, 13 chars)
Tokens: ["token", "##ization"]         (2 tokens)

Word:   "hello"                 (1 word, 5 chars)
Tokens: ["hello"]                      (1 token — common word)

Word:   " unbelievable"         (GPT-2 style, leading space)
Tokens: ["Ġun", "believ", "able"]      (3 tokens — different split)

Modern tokenizers use subword algorithms (BPE, WordPiece, SentencePiece). The core principle: build a vocabulary of pieces that balances coverage and sequence length.

• Common words stay whole because they appear frequently enough to earn their own vocabulary slot.
• Rare, long, or compound words split into reusable pieces rather than being marked [UNK].
• Punctuation often becomes its own token — commas, periods, hyphens, and quotes are typically isolated.
• Whitespace handling differs by tokenizer: GPT-2 style tokenizers include the leading space as part of the token (e.g., Ġplaying). BERT uses ## prefixes on continuation pieces.

Take this sentence and manually estimate the word count vs token count:

Sentence: "The tokenizer is surprisingly useful."

Step 1 — Count words (split on spaces):
  ["The", "tokenizer", "is", "surprisingly", "useful."]
  Word count = 5

Step 2 — Estimate token count (BERT-style):
  "The"          → ["the"]                     = 1 token
  "tokenizer"    → ["token", "##izer"]         = 2 tokens
  "is"           → ["is"]                      = 1 token
  "surprisingly" → ["surpris", "##ingly"]      = 2 tokens
  "useful"       → ["useful"]                  = 1 token
  "."            → ["."]                       = 1 token
                                         Total = 8 tokens

Ratio: 8 tokens / 5 words = 1.6 tokens per word
(Higher than average because "tokenizer" and "surprisingly" are rare)

The token/word ratio τ tells you how many tokens you get per word. It varies dramatically by text type, language, and tokenizer.

Embedding Weight Equation

For any language model, the embedding matrix size is:

Where $V$ is vocabulary size and $d_{\text{model}}$ is the hidden dimension. For a LLaMA model ($d_{\text{model}} = 4096$) in FP16 (2 bytes/weight):

Conclusion: A whole-word vocabulary wastes over 8 GB of GPU VRAM on static lookup tables before a single attention operation runs. Capping at ~32K subwords keeps the footprint tiny.

Type or paste text below. Each word is counted by whitespace splitting. Each token is estimated with a heuristic: words longer than 8 characters are treated as likely 2 tokens; punctuation gets isolated. Colors show individual token spans.

graph TD
    subgraph WordLevel [Word-Level Paradigm]
        W["'unbelievingly'"] --> W_ID["[ID: 981244]  ❌ Fails if unseen → UNK"]
    end
    subgraph TokenLevel [Subword Token Paradigm]
        T["'unbelievingly'"] --> T1["'un'  (ID: 451)"]
        T --> T2["'##believ'  (ID: 8711)"]
        T --> T3["'##ingly'  (ID: 1042)"]
    end
    style W fill:#ef4444,stroke:#dc2626,color:#fff
    style T1 fill:#10b981,stroke:#059669,color:#fff
    style T2 fill:#10b981,stroke:#059669,color:#fff
    style T3 fill:#10b981,stroke:#059669,color:#fff
        

If the model has never seen "unbelievingly", the word-level system yields [UNK]. The subword system falls back to "un" + "##believ" + "##ingly" — all pieces it already understands from other contexts.

Run both approaches and compare. The gap is the lesson.

text = "The tokenizer is surprisingly useful."

# ── Approach 1: naive word split ──
words = text.split()
print(f"Words:      {words}")
print(f"Word count: {len(words)}")

# ── Approach 2: real tokenizer (BERT) ──
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
tokens = tok.tokenize(text)
print(f"\nTokens:      {tokens}")
print(f"Token count: {len(tokens)}")

# ── Ratio ──
ratio = len(tokens) / len(words)
print(f"\nToken/word ratio τ = {ratio:.2f}")

# ── Expected output ──
# Words:      ['The', 'tokenizer', 'is', 'surprisingly', 'useful.']
# Word count: 5
#
# Tokens:      ['the', 'token', '##izer', 'is', 'surpris', '##ingly', 'useful', '.']
# Token count: 8
#
# Token/word ratio τ = 1.60

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
text = "The tokenizer is surprisingly useful."

token_ids = enc.encode(text)
word_count = len(text.split())

print(f"Token IDs:   {token_ids}")
print(f"Token count: {len(token_ids)}")
print(f"Word count:  {word_count}")
print(f"τ = {len(token_ids) / word_count:.2f}")
# GPT-4: 5 words → ~7 tokens (τ ≈ 1.40)
# Larger vocab means more words fit in 1 token vs BERT

Run a terminal simulation feeding a typo-ridden sentence to both tokenizer types to observe noise tolerance. Then complete the hands-on checklist.

⚠️ Common Beginner Misconceptions

• "Tokens are split on syllables." False. Tokenizers use statistical pair frequencies (BPE), not grammatical rules. That's why "therapy" can split into ["the", "rapy"].
• "Spaces are discarded." False. Spacing is critical for generation. Tokenizers treat space-character transitions as explicit characters — " word" and "word" are completely different tokens.
• "1 word = 1 token in modern LLMs." False. GPT-4's vocabulary is larger so common words often get 1 token — but rare, technical, or compound words still split.

Quiz: What is the most accurate description of the difference between a word and a token?

📌 Summary — what to remember

• Words ≠ tokens: A word is a human unit; a token is the model's unit. One word can produce multiple tokens.
• Typical ratio τ: 1.3–1.5 for English prose, higher for technical text, code, and non-Latin scripts.
• Why it matters: API pricing, context window limits, and RAG chunk sizing all operate in tokens. Word-count budgets are systematically wrong.
• Punctuation and numbers: Punctuation marks typically become their own token. Rare number sequences often split per digit.
• Tokenizer-specific: BERT, GPT-2, and LLaMA tokenizers produce different counts for the same text. Never assume counts transfer between models.
• Practical rule: Always tokenize before sending API calls. Use tiktoken for GPT models, AutoTokenizer for HuggingFace models.