Characters vs Tokens

Before a neural network can process human language, continuous strings must be broken down into discrete units called tokens and assigned numerical identifiers. When designing a tokenizer, engineers face three fundamental choices of granularity:

Both extreme approaches have severe, disqualifying failure modes that led researchers to create statistical subword algorithms like BPE and WordPiece:

Think of constructing a written sentence like assembling a structure out of building blocks. The core design question is: what size should the standard building blocks be?

Every tokenization architecture is a point on a continuous mathematical trade-off curve between vocabulary size ($V$), sequence length ($N$), and OOV coverage.

Let's trace exactly how the phrase "playing well" is split across all three paradigms, showing the exact tokens, representations, and boundaries:

graph TD
    Text["Input: 'playing well'"]
    
    subgraph Char["Character Level (N = 12)"]
        direction LR
        C1["p"] --> C2["l"] --> C3["a"] --> C4["y"] --> C5["i"] --> C6["n"] --> C7["g"] --> C8["␣"] --> C9["w"] --> C10["e"] --> C11["l"] --> C12["l"]
    end
    
    subgraph Word["Word Level (N = 2)"]
        direction LR
        W1["playing"] --> W2["well"]
    end
    
    subgraph Sub["Subword Level (N = 3)"]
        direction LR
        S1["play"] --> S2["##ing"] --> S3["well"]
    end
    
    Text --> Char
    Text --> Word
    Text --> Sub

    style C1 fill:#ef4444,stroke:#dc2626,color:#fff
    style C2 fill:#ef4444,stroke:#dc2626,color:#fff
    style C3 fill:#ef4444,stroke:#dc2626,color:#fff
    style C4 fill:#ef4444,stroke:#dc2626,color:#fff
    style C5 fill:#ef4444,stroke:#dc2626,color:#fff
    style C6 fill:#ef4444,stroke:#dc2626,color:#fff
    style C7 fill:#ef4444,stroke:#dc2626,color:#fff
    style C8 fill:#ef4444,stroke:#dc2626,color:#fff
    style C9 fill:#ef4444,stroke:#dc2626,color:#fff
    style C10 fill:#ef4444,stroke:#dc2626,color:#fff
    style C11 fill:#ef4444,stroke:#dc2626,color:#fff
    style C12 fill:#ef4444,stroke:#dc2626,color:#fff
    style W1 fill:#10b981,stroke:#059669,color:#fff
    style W2 fill:#10b981,stroke:#059669,color:#fff
    style S1 fill:#3b82f6,stroke:#2563eb,color:#fff
    style S2 fill:#3b82f6,stroke:#2563eb,color:#fff
    style S3 fill:#3b82f6,stroke:#2563eb,color:#fff
          

Not all text tokenizes cleanly into dictionary words. Here are three critical edge cases analyzed through rigorous mathematical steps:

Compare how different content types expand under different tokenization approaches:

Run real tokenization experiments comparing raw character lists, whitespace splits, and Hugging Face's BERT WordPiece tokenizer:

from transformers import AutoTokenizer

# Load BERT WordPiece tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

sample_texts = [
    "playing well",
    "unbelievable",
    "def calculate_total(user_id):",
    "2024"
]

print(f"{'Input Text':<30} | {'Chars':<6} | {'Words':<6} | {'Subwords':<8} | Subword Tokens")
print("-" * 80)

for text in sample_texts:
    char_list = list(text)
    word_list = text.split()
    subword_tokens = tokenizer.tokenize(text)
    
    print(f"{text:<30} | {len(char_list):<6} | {len(word_list):<6} | {len(subword_tokens):<8} | {subword_tokens}")

A comprehensive side-by-side comparison across all 4 production tokenization paradigms:

Understanding the character-vs-token ratio is crucial for calculating prompt token budgets, API costs, and context window limits in production LLM applications.

• Character ≠ Byte: In UTF-8, English letters take 1 byte, European accents take 2 bytes, Asian characters take 3 bytes, and emojis take 4 bytes. Byte-level tokenizers operate on raw bytes, not Unicode code points.
• Subwords Are Not Linguistic Morphemes: Tokenizers are trained purely on unsupervised statistical frequency, not linguistic rules. The pieces un + ##believ + ##able are statistical co-occurrences, not formal grammatical parses.
• Attention Quadratic Cost is Inescapable: A 4× increase in sequence length yields a 16× increase in self-attention compute ($O(N^2)$). Character models cannot scale to 128k context windows without massive hierarchical downsampling.
• Token Count Bias (The Multilingual Tax): Because tokenizers are trained predominantly on English text, non-English speakers pay 3× to 8× more API cost for the exact same semantic meaning.

⚠️ Common Beginner Misconceptions

• "Character-level is better because the model size is smaller." False. While the embedding matrix is smaller, the self-attention memory and compute blow up by $16\times$ to $25\times$, bottlenecking training and inference throughput.
• "Tokenizers understand words before characters." False. Neural networks have zero intrinsic concept of letters or words — they only see sequential integer token IDs like [15234, 492] and learn statistical relations through attention weights.