zations|🎮 Interactive Tools & Visualizations]]

• 🎉 Fun Facts & Hacks
  • 🤯 Mind-Blowing Facts
  • 💰 Cost-Saving Hacks
  • 🎯 Practical Hacks
• 🎯 What You'll Learn
• 🧠 Core Concept
  • What is a Token?
  • Why Not Just Use Words?
  • The Solution: Subword Tokenization
• 🎨 Tokenization Pipeline Visualization
  • Step-by-Step Breakdown
  • Algorithm Decision Tree
  • BPE Merge Process Visualization
  • Tokenization Comparison Flow
• 💻 Code Examples
  • Example 1: Simple Word Tokenization (Basic)
  • Example 2: Character-Level Tokenization
  • Example 3: Byte Pair Encoding (BPE) - From Scratch
  • Example 4: Using HuggingFace Tokenizers (Production-Ready)
  • Example 5: Practical Token Counting for API Costs
• 🔬 Advanced Tokenization Algorithms
  • Algorithm Comparison Table
  • 🎯 Quick Comparison
  • WordPiece Tokenization (BERT, DistilBERT)
    • How WordPiece Works
    • WordPiece vs BPE
    • WordPiece Tokenization Process
    • Key Features
    • Real-World Example: BERT Tokenization
  • SentencePiece (T5, XLNet, ALBERT)
    • Why SentencePiece?
    • SentencePiece Architecture
    • Key Features
    • SentencePiece Training Process
  • Unigram Language Model (XLNet, ALBERT)
    • How Unigram LM Works
    • Unigram vs BPE/WordPiece
    • Probabilistic Tokenization
    • EM Algorithm for Training
    • Unigram LM Advantages
  • Example 6: WordPiece Tokenization Implementation
  • Example 7: SentencePiece for Multilingual Text
  • Production Use Cases
    • BERT (WordPiece)
    • T5 (SentencePiece)
    • XLNet (SentencePiece + Unigram)
• 🌍 Multilingual Tokenization
  • The Multilingual Challenge
  • Unicode Normalization and Handling
    • Unicode Normalization Forms
    • Four Normalization Forms
    • Unicode Categories
  • Language-Specific Challenges
    • 1. CJK Languages (Chinese, Japanese, Korean)
    • 2. Arabic and RTL Languages
    • 3. Indic Scripts (Hindi, Tamil, Bengali, etc.)
  • Example 8: Handling Multilingual Text
  • Best Practices for Multilingual Tokenization
    • 1. Always Normalize Unicode
    • 2. Use Language-Agnostic Tools
    • 3. Handle Special Cases
    • 4. Test with Real Data
  • Common Pitfalls
• 🎯 Hands-On Exercise
  • Exercise 1: Build a Custom Tokenizer
  • Exercise 2: Token Efficiency Analysis
• 🔍 Common Pitfalls & Debugging
  • Pitfall 1: Token Limits
  • Pitfall 2: Inconsistent Tokenization
  • Pitfall 3: Special Characters
• 🔗 Related Concepts
• 📚 Resources
  • Essential Reading
  • Video Tutorials
  • Interactive Tools
  • Code Libraries
• 📝 Assignments & Practice Problems
  • Assignment 1: Build a Character-Level Tokenizer ⭐
  • Assignment 2: Implement BPE Merge Operation ⭐⭐
  • Assignment 3: Token Cost Analyzer ⭐⭐
  • Assignment 4: Multi-Language Tokenizer ⭐⭐⭐
  • Assignment 5: Tokenization Benchmark Suite ⭐⭐⭐
• 🎯 Assignment Submission Checklist
• 💡 Learning Outcomes
• 🆘 Getting Help
• ✅ Key Takeaways
• 🎯 Next Steps

🔤 Tokenization

🌟 Start With a Real-World Scenario

Imagine this: You're texting your friend: "I'll meet you at 5pm tomorrow! 😊"

Your friend reads it instantly and understands everything. But here's the thing—AI can't read English!

When you send that message to ChatGPT, it doesn't see words. It sees this:

["I", "'ll", " meet", " you", " at", " ", "5", "pm", " tomorrow", "!", " ", "😊"]

Or even stranger, it might see:

[40, 3278, 1826, 345, 379, 642, 4426, 0, 50118]

Wait, what? 🤔

That's tokenization! Before AI can understand your message, it must:

1. Break it down into pieces (tokens)
2. Convert each piece into numbers
3. Process those numbers through its neural network

Here's the crazy part: The way AI breaks down your text affects:

• ✅ How well it understands you
• 💰 How much you pay (APIs charge per token!)
• ⚡ How fast it responds
• 🎯 Whether it can handle your language/emojis

Real Example:

• "ChatGPT" = 2 tokens = $0.00004 (with GPT-4)
• "Chat GPT" = 3 tokens = $0.00006 (50% more expensive!)
• "🤖" = 1-3 tokens depending on the model

By the end of this guide, you'll understand:

• Why "don't" costs less than "do not"
• How to save 30-50% on API costs
• Why AI sometimes cuts off mid-sentence (token limits!)
• How to build your own tokenizer from scratch

Let's dive in! 🚀

🎨 Use This Structure

Here's what to include in your mind map:

                    TOKENIZATION
                         |
        +----------------+----------------+
        |                |                |
    WORD-LEVEL      SUBWORD-LEVEL    CHARACTER-LEVEL
        |                |                |
    - Simple         - BPE            - Flexible
    - Large vocab    - WordPiece      - Huge vocab
    - OOV issues     - SentencePiece  - Long sequences
                         |
                    +----+----+
                    |         |
                 GPT-2      BERT
                (BPE)    (WordPiece)

🧠 Core Concept

What is a Token?

A token is a unit of text that a model processes. It could be:

• A word: "hello" → ["hello"]
• A subword: "unhappiness" → ["un", "happiness"]
• A character: "cat" → ["c", "a", "t"]
• A byte: Used in byte-level tokenization

Why Not Just Use Words?

Problem 1: Vocabulary Size

# English has ~170,000 words
# Add technical terms, names, slang → millions of words
# Each word needs a unique ID → huge memory requirement

Problem 2: Unknown Words

# What if the model sees "ChatGPT" but was trained before 2022?
# Word-level: <UNK> (unknown token) → loses meaning
# Subword-level: ["Chat", "G", "PT"] → preserves meaning

Problem 3: Morphology

# "run", "running", "runs", "ran" are related
# Word-level: 4 separate tokens
# Subword-level: ["run", "##ning", "##s"] → shares "run" embedding

The Solution: Subword Tokenization

🎨 Tokenization Pipeline Visualization

Let's visualize how text flows through a tokenization pipeline:

graph LR
    A[Raw Text] --> B[Normalization]
    B --> C[Pre-tokenization]
    C --> D[Tokenization Algorithm]
    D --> E[Post-processing]
    E --> F[Token IDs]
    
    style A fill:#e1f5ff
    style B fill:#fff9c4
    style C fill:#f8bbd0
    style D fill:#c5e1a5
    style E fill:#ffccbc
    style F fill:#b2dfdb

Step-by-Step Breakdown

1. Raw Text 📝

"Hello, world! 🌍"

2. Normalization 🔄

• Lowercase conversion (optional)
• Unicode normalization (NFD/NFC)
• Accent removal (optional)

"hello, world! 🌍"

3. Pre-tokenization ✂️

• Split on whitespace
• Handle punctuation

["hello", ",", "world", "!", "🌍"]

4. Tokenization Algorithm 🧮

• Apply BPE/WordPiece/SentencePiece
• Break into subwords if needed

["hello", ",", "world", "!", "🌍"]  # Already optimal

5. Post-processing 🎯

• Add special tokens ([CLS], [SEP])
• Create attention masks

["[CLS]", "hello", ",", "world", "!", "🌍", "[SEP]"]

6. Token IDs 🔢

• Convert to numerical IDs

[101, 7592, 1010, 2088, 999, 127757, 102]

Algorithm Decision Tree

Which tokenization algorithm should you use? Here's a decision tree:

graph TD
    A[Choose Tokenization Algorithm] --> B{What's your use case?}
    B -->|Simple English text| C[BPE]
    B -->|BERT-style models| D[WordPiece]
    B -->|Multilingual| E[SentencePiece]
    B -->|Research/Custom| F[Unigram LM]
    
    C --> G[GPT-2, GPT-3, RoBERTa]
    D --> H[BERT, DistilBERT]
    E --> I[T5, XLNet, ALBERT, mT5]
    F --> J[XLNet, ALBERT]
    
    style A fill:#667eea
    style B fill:#f093fb
    style C fill:#4facfe
    style D fill:#43e97b
    style E fill:#fa709a
    style F fill:#fee140

BPE Merge Process Visualization

Here's how Byte Pair Encoding merges tokens:

graph TD
    A["Initial: ['l', 'o', 'w', 'e', 'r']"] --> B["Count pairs: 'lo'=5, 'ow'=3, 'we'=2, 'er'=4"]
    B --> C["Merge most frequent: 'er' → 'er'"]
    C --> D["New: ['l', 'o', 'w', 'er']"]
    D --> E["Count pairs: 'lo'=5, 'ow'=3, 'wer'=2"]
    E --> F["Merge: 'lo' → 'lo'"]
    F --> G["Final: ['lo', 'w', 'er']"]
    
    style A fill:#e1f5ff
    style G fill:#c5e1a5

Tokenization Comparison Flow

graph LR
    A["Input: 'unhappiness'"] --> B[Word-Level]
    A --> C[Character-Level]
    A --> D[Subword BPE]
    A --> E[WordPiece]
    
    B --> B1["['unhappiness']<br/>1 token"]
    C --> C1["['u','n','h','a','p','p','i','n','e','s','s']<br/>11 tokens"]
    D --> D1["['un','happiness']<br/>2 tokens"]
    E --> E1["['un','##happiness']<br/>2 tokens"]
    
    style A fill:#667eea
    style B1 fill:#4facfe
    style C1 fill:#fa709a
    style D1 fill:#43e97b
    style E1 fill:#fee140

Modern tokenizers use subword units to balance vocabulary size and coverage:

# Example: GPT-3 uses ~50,000 tokens
# Can represent millions of words through combinations
"unbelievable" → ["un", "believ", "able"]
"ChatGPT" → ["Chat", "G", "PT"]
"🚀" → ["🚀"]  # Emojis are often single tokens

💻 Code Examples

Example 1: Simple Word Tokenization (Basic)

def simple_word_tokenizer(text):
    """
    Basic word tokenization using whitespace splitting.
    Good for understanding, not for production.
    """
    # Convert to lowercase and split by spaces
    tokens = text.lower().split()
    
    # Remove punctuation (simple approach)
    import string
    tokens = [token.strip(string.punctuation) for token in tokens]
    
    return tokens

# Test it
text = "Hello, world! How are you doing today?"
tokens = simple_word_tokenizer(text)
print(f"Tokens: {tokens}")
print(f"Token count: {len(tokens)}")

# Output:
# Tokens: ['hello', 'world', 'how', 'are', 'you', 'doing', 'today']
# Token count: 7

Limitations:

• Can't handle contractions: "don't" → "dont"
• Loses punctuation information
• No handling of unknown words
• Huge vocabulary needed

Example 2: Character-Level Tokenization

def character_tokenizer(text):
    """
    Character-level tokenization.
    Small vocabulary but long sequences.
    """
    # Each character is a token
    tokens = list(text)
    
    # Create vocabulary
    vocab = sorted(set(tokens))
    
    # Create token-to-id mapping
    token_to_id = {token: idx for idx, token in enumerate(vocab)}
    
    # Convert tokens to IDs
    token_ids = [token_to_id[token] for token in tokens]
    
    return tokens, token_ids, vocab

# Test it
text = "Hello!"
tokens, token_ids, vocab = character_tokenizer(text)

print(f"Text: {text}")
print(f"Tokens: {tokens}")
print(f"Token IDs: {token_ids}")
print(f"Vocabulary: {vocab}")
print(f"Vocab size: {len(vocab)}")

# Output:
# Text: Hello!
# Tokens: ['H', 'e', 'l', 'l', 'o', '!']
# Token IDs: [0, 3, 4, 4, 5, 1]
# Vocabulary: ['!', 'H', 'e', 'l', 'o']
# Vocab size: 5

Pros:

• ✅ Tiny vocabulary (26 letters + punctuation)
• ✅ No unknown tokens
• ✅ Works for any language

Cons:

• ❌ Very long sequences (1 char = 1 token)
• ❌ Model must learn word meanings from scratch
• ❌ Computationally expensive

Example 3: Byte Pair Encoding (BPE) - From Scratch

from collections import Counter
import re

class SimpleBPETokenizer:
    """
    Simplified BPE tokenizer implementation.
    This is how GPT-2, GPT-3, and many modern models tokenize.
    """
    
    def __init__(self, vocab_size=1000):
        self.vocab_size = vocab_size
        self.vocab = {}
        self.merges = {}
        
    def get_stats(self, words):
        """Count frequency of adjacent token pairs."""
        pairs = Counter()
        for word, freq in words.items():
            symbols = word.split()
            for i in range(len(symbols) - 1):
                pairs[(symbols[i], symbols[i + 1])] += freq
        return pairs
    
    def merge_vocab(self, pair, words):
        """Merge the most frequent pair in vocabulary."""
        new_words = {}
        bigram = ' '.join(pair)
        replacement = ''.join(pair)
        
        for word in words:
            new_word = word.replace(bigram, replacement)
            new_words[new_word] = words[word]
        return new_words
    
    def train(self, text):
        """Train BPE on text corpus."""
        # Start with character-level tokens
        words = {}
        for word in text.split():
            # Add space between characters
            word_tokens = ' '.join(list(word)) + ' </w>'
            words[word_tokens] = words.get(word_tokens, 0) + 1
        
        # Iteratively merge most frequent pairs
        for i in range(self.vocab_size - 256):  # 256 for base characters
            pairs = self.get_stats(words)
            if not pairs:
                break
            
            # Get most frequent pair
            best_pair = max(pairs, key=pairs.get)
            words = self.merge_vocab(best_pair, words)
            self.merges[best_pair] = i
            
            if i % 100 == 0:
                print(f"Merge {i}: {best_pair} (freq: {pairs[best_pair]})")
        
        # Build vocabulary
        self.vocab = set()
        for word in words:
            self.vocab.update(word.split())
        
        print(f"\nTraining complete!")
        print(f"Vocabulary size: {len(self.vocab)}")
        print(f"Number of merges: {len(self.merges)}")
    
    def tokenize(self, text):
        """Tokenize text using learned BPE merges."""
        tokens = []
        for word in text.split():
            # Start with characters
            word_tokens = list(word) + ['</w>']
            
            # Apply merges
            while len(word_tokens) > 1:
                pairs = [(word_tokens[i], word_tokens[i + 1]) 
                        for i in range(len(word_tokens) - 1)]
                
                # Find pair with lowest merge index (earliest merge)
                valid_pairs = [p for p in pairs if p in self.merges]
                if not valid_pairs:
                    break
                
                best_pair = min(valid_pairs, key=lambda p: self.merges[p])
                
                # Merge the pair
                new_tokens = []
                i = 0
                while i < len(word_tokens):
                    if i < len(word_tokens) - 1 and \
                       (word_tokens[i], word_tokens[i + 1]) == best_pair:
                        new_tokens.append(word_tokens[i] + word_tokens[i + 1])
                        i += 2
                    else:
                        new_tokens.append(word_tokens[i])
                        i += 1
                word_tokens = new_tokens
            
            tokens.extend(word_tokens)
        
        return tokens

# Train the tokenizer
corpus = """
The quick brown fox jumps over the lazy dog.
The dog was not amused by the fox's antics.
Quick thinking saved the day.
"""

tokenizer = SimpleBPETokenizer(vocab_size=300)
tokenizer.train(corpus)

# Test tokenization
test_text = "The quick fox"
tokens = tokenizer.tokenize(test_text)
print(f"\nTokenizing: '{test_text}'")
print(f"Tokens: {tokens}")

Output:

Merge 0: ('e', ' ') (freq: 8)
Merge 100: ('t', 'h') (freq: 5)
...
Training complete!
Vocabulary size: 287
Number of merges: 44

Tokenizing: 'The quick fox'
Tokens: ['The</w>', 'qu', 'ick</w>', 'fox</w>']

Example 4: Using HuggingFace Tokenizers (Production-Ready)

from transformers import AutoTokenizer

# Load pre-trained tokenizers
gpt2_tokenizer = AutoTokenizer.from_pretrained("gpt2")
bert_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "Hello, how are you doing today? I'm learning about tokenization!"

# GPT-2 tokenization (BPE)
print("=== GPT-2 Tokenizer ===")
gpt2_tokens = gpt2_tokenizer.tokenize(text)
gpt2_ids = gpt2_tokenizer.encode(text)
print(f"Tokens: {gpt2_tokens}")
print(f"Token IDs: {gpt2_ids}")
print(f"Token count: {len(gpt2_tokens)}")

# BERT tokenization (WordPiece)
print("\n=== BERT Tokenizer ===")
bert_tokens = bert_tokenizer.tokenize(text)
bert_ids = bert_tokenizer.encode(text)
print(f"Tokens: {bert_tokens}")
print(f"Token IDs: {bert_ids}")
print(f"Token count: {len(bert_tokens)}")

# Decode back to text
decoded_text = gpt2_tokenizer.decode(gpt2_ids)
print(f"\nDecoded: {decoded_text}")

Output:

=== GPT-2 Tokenizer ===
Tokens: ['Hello', ',', 'Ġhow', 'Ġare', 'Ġyou', 'Ġdoing', 'Ġtoday', '?', 'ĠI', "'m", 'Ġlearning', 'Ġabout', 'Ġtoken', 'ization', '!']
Token IDs: [15496, 11, 703, 389, 345, 1804, 1909, 30, 314, 1101, 4673, 546, 11241, 1634, 0]
Token count: 15

=== BERT Tokenizer ===
Tokens: ['hello', ',', 'how', 'are', 'you', 'doing', 'today', '?', 'i', "'", 'm', 'learning', 'about', 'token', '##ization', '!']
Token IDs: [101, 7592, 1010, 2129, 2024, 2017, 2725, 2651, 1029, 1045, 1005, 1049, 4083, 2055, 19204, 3989, 999, 102]
Token count: 16

Decoded: Hello, how are you doing today? I'm learning about tokenization!

Key Observations:

• GPT-2 uses Ġ to indicate spaces
• BERT uses ## for subword continuations
• Different tokenizers = different token counts
• Special tokens: [CLS] (101), [SEP] (102) in BERT

Example 5: Practical Token Counting for API Costs

import tiktoken  # OpenAI's tokenizer

def count_tokens(text, model="gpt-4"):
    """
    Count tokens for OpenAI API pricing.
    GPT-4: $0.03 per 1K input tokens, $0.06 per 1K output tokens
    """
    encoding = tiktoken.encoding_for_model(model)
    tokens = encoding.encode(text)
    return len(tokens)

# Example: Estimate API costs
prompt = """
You are a helpful AI assistant. Please analyze the following code
and suggest improvements:

def calculate_sum(numbers):
    total = 0
    for num in numbers:
        total = total + num
    return total
"""

token_count = count_tokens(prompt, model="gpt-4")
input_cost = (token_count / 1000) * 0.03

print(f"Prompt: {prompt[:50]}...")
print(f"Token count: {token_count}")
print(f"Estimated input cost: ${input_cost:.4f}")
print(f"\nTip: Shorter prompts = lower costs!")

# Compare different phrasings
short_prompt = "Analyze this code and suggest improvements:\n\n" + \
               "def calculate_sum(numbers):\n    return sum(numbers)"

short_tokens = count_tokens(short_prompt, model="gpt-4")
short_cost = (short_tokens / 1000) * 0.03

print(f"\n=== Optimized Version ===")
print(f"Token count: {short_tokens}")
print(f"Estimated input cost: ${short_cost:.4f}")
print(f"Savings: {token_count - short_tokens} tokens ({((token_count - short_tokens) / token_count * 100):.1f}%)")

🔬 Advanced Tokenization Algorithms

Now that you understand the basics, let's explore the advanced tokenization algorithms used in production AI systems!

Algorithm Comparison Table

🎯 Quick Comparison

graph TD
    A[Tokenization Algorithms] --> B[BPE]
    A --> C[WordPiece]
    A --> D[SentencePiece]
    A --> E[Unigram LM]
    
    B --> B1[Merge frequent pairs<br/>Character-level start]
    C --> C1[Greedy longest match<br/>Likelihood scoring]
    D --> D1[Language-agnostic<br/>Raw byte processing]
    E --> E1[Probabilistic<br/>EM algorithm]
    
    style A fill:#667eea
    style B fill:#4facfe
    style C fill:#43e97b
    style D fill:#fa709a
    style E fill:#fee140

WordPiece Tokenization (BERT, DistilBERT)

Used by: BERT, DistilBERT, Electra

Key Idea: Instead of merging the most frequent pair (like BPE), WordPiece merges pairs that maximize the likelihood of the training data.

How WordPiece Works

1. Start with characters (like BPE)
2. Score each merge by how much it increases data likelihood
3. Greedily select the best merge
4. Repeat until vocabulary size reached

WordPiece vs BPE

# BPE: Merge most frequent pair
# "low" appears 5 times, "est" appears 3 times
# → Merge "low" first (higher frequency)

# WordPiece: Merge pair with highest likelihood score
# Score("low") = P("low") / (P("l") * P("ow"))
# Score("est") = P("est") / (P("e") * P("st"))
# → Merge whichever has higher score (not just frequency)

WordPiece Tokenization Process

graph LR
    A["Input: 'unhappiness'"] --> B[Split into chars]
    B --> C["['u','n','h','a','p','p','i','n','e','s','s']"]
    C --> D[Greedy longest match]
    D --> E{Is 'unhappiness'<br/>in vocab?}
    E -->|No| F{Is 'unhappi'<br/>in vocab?}
    F -->|No| G{Is 'un'<br/>in vocab?}
    G -->|Yes| H["Add 'un'"]
    H --> I{Is 'happiness'<br/>in vocab?}
    I -->|Yes| J["Add '##happiness'"]
    J --> K["Result: ['un', '##happiness']"]
    
    style A fill:#e1f5ff
    style K fill:#c5e1a5

Key Features

1. Subword Prefix: Uses ## to indicate continuation

"unhappiness" → ["un", "##happiness"]
"playing" → ["play", "##ing"]

2. Greedy Longest Match: Always tries to match the longest possible subword first

# If vocabulary has: ["un", "happy", "happiness", "##ness"]
"unhappiness" → ["un", "##happiness"]  # Not ["un", "##happy", "##ness"]

3. Unknown Handling: Falls back to characters if needed

"xyz123" → ["x", "##y", "##z", "##1", "##2", "##3"]

Real-World Example: BERT Tokenization

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Example 1: Common words
text1 = "The quick brown fox"
tokens1 = tokenizer.tokenize(text1)
print(f"'{text1}' → {tokens1}")
# Output: ['the', 'quick', 'brown', 'fox']

# Example 2: Rare/compound words
text2 = "unhappiness"
tokens2 = tokenizer.tokenize(text2)
print(f"'{text2}' → {tokens2}")
# Output: ['un', '##hap', '##pi', '##ness']

# Example 3: Technical terms
text3 = "COVID-19 pandemic"
tokens3 = tokenizer.tokenize(text3)
print(f"'{text3}' → {tokens3}")
# Output: ['co', '##vid', '-', '19', 'pandemic']

# See token IDs
ids = tokenizer.encode(text1)
print(f"Token IDs: {ids}")
# Output: [101, 1996, 4248, 2829, 4419, 102]
#         [CLS] the  quick brown fox  [SEP]

SentencePiece (T5, XLNet, ALBERT)

Used by: T5, XLNet, ALBERT, mT5, mBART

Key Idea: Treat text as a raw byte stream (no pre-tokenization), making it truly language-agnostic!

Why SentencePiece?

Problem with BPE/WordPiece:

# English: Pre-tokenize on spaces
"Hello world" → ["Hello", "world"] → tokenize each

# Chinese: No spaces!
"你好世界" → ??? → Can't pre-tokenize on spaces

# Japanese: Mixed scripts
"こんにちは世界" → ??? → Complex segmentation needed

SentencePiece Solution:

# Treat ALL text as raw bytes
"Hello world" → "▁Hello▁world" → tokenize
"你好世界" → "▁你好世界" → tokenize
# ▁ represents space (U+2581)

SentencePiece Architecture

graph TD
    A[Raw Text] --> B[No Pre-tokenization]
    B --> C[Treat as Byte Stream]
    C --> D{Algorithm Choice}
    D -->|Option 1| E[Unigram LM]
    D -->|Option 2| F[BPE]
    E --> G[Vocabulary]
    F --> G
    G --> H[Tokenize]
    H --> I[Tokens with ▁]
    
    style A fill:#e1f5ff
    style C fill:#fff9c4
    style G fill:#c5e1a5
    style I fill:#b2dfdb

Key Features

1. Language-Agnostic: Works for ANY language

# English
"Hello world" → ["▁Hello", "▁world"]

# Chinese (no spaces)
"你好世界" → ["▁你好", "世界"]

# Arabic (RTL)
"مرحبا" → ["▁مرحبا"]

# Mixed
"Hello 世界" → ["▁Hello", "▁世", "界"]

2. Reversible: Can perfectly reconstruct original text

tokens = ["▁Hello", "▁world", "!"]
text = "".join(tokens).replace("▁", " ").strip()
# Result: "Hello world!"

3. No Vocabulary Dependency: Doesn't need language-specific rules

SentencePiece Training Process

import sentencepiece as spm

# Step 1: Train a SentencePiece model
spm.SentencePieceTrainer.train(
    input='corpus.txt',           # Your training data
    model_prefix='my_tokenizer',  # Output model name
    vocab_size=32000,             # Vocabulary size
    model_type='unigram',         # or 'bpe'
    character_coverage=0.9995,    # Cover 99.95% of characters
    pad_id=0,                     # Padding token ID
    unk_id=1,                     # Unknown token ID
    bos_id=2,                     # Beginning of sentence
    eos_id=3,                     # End of sentence
)

# Step 2: Load and use the model
sp = spm.SentencePieceProcessor()
sp.load('my_tokenizer.model')

# Tokenize
text = "Hello world! 你好世界"
tokens = sp.encode_as_pieces(text)
print(f"Tokens: {tokens}")
# Output: ['▁Hello', '▁world', '!', '▁你好', '世界']

# Get IDs
ids = sp.encode_as_ids(text)
print(f"IDs: {ids}")

# Detokenize
reconstructed = sp.decode_pieces(tokens)
print(f"Reconstructed: {reconstructed}")
# Output: "Hello world! 你好世界"

Unigram Language Model (XLNet, ALBERT)

Used by: XLNet, ALBERT (via SentencePiece)

Key Idea: Use a probabilistic approach to select the best tokenization, not just greedy merging.

How Unigram LM Works

Unlike BPE (bottom-up) or WordPiece (greedy), Unigram LM is top-down:

1. Start with a large vocabulary (all possible substrings)
2. Assign probabilities to each subword
3. Iteratively remove subwords that least affect likelihood
4. Stop when vocabulary reaches target size

Unigram vs BPE/WordPiece

graph TD
    A[Tokenization Approaches] --> B[BPE/WordPiece]
    A --> C[Unigram LM]
    
    B --> B1[Bottom-up<br/>Start with characters<br/>Merge upward]
    C --> C1[Top-down<br/>Start with all substrings<br/>Prune downward]
    
    B --> B2[Deterministic<br/>One tokenization]
    C --> C2[Probabilistic<br/>Multiple possible<br/>tokenizations]
    
    style A fill:#667eea
    style B fill:#4facfe
    style C fill:#fa709a

Probabilistic Tokenization

Key Insight: For a given text, there are multiple valid tokenizations!

# Text: "unhappiness"

# Possible tokenizations:
# 1. ["un", "happiness"]     P = 0.45
# 2. ["un", "happy", "ness"] P = 0.30
# 3. ["unhap", "piness"]     P = 0.15
# 4. ["u", "n", "happiness"] P = 0.10

# Unigram LM: Choose based on probability!
# Usually picks #1 (highest probability)

EM Algorithm for Training

# Expectation-Maximization (EM) Algorithm

# E-Step: Given current vocabulary, compute best tokenization
for text in corpus:
    tokenizations = get_all_possible_tokenizations(text, vocab)
    probabilities = [compute_probability(t, vocab) for t in tokenizations]
    best_tokenization = tokenizations[argmax(probabilities)]

# M-Step: Update vocabulary probabilities
for subword in vocab:
    count = count_occurrences(subword, all_tokenizations)
    probability[subword] = count / total_count

# Prune: Remove least useful subwords
vocab = remove_low_probability_subwords(vocab, target_size)

Unigram LM Advantages

1. Multiple Tokenizations: Can sample different tokenizations

# Same text, different tokenizations (data augmentation!)
"unhappiness" → ["un", "happiness"]
"unhappiness" → ["un", "happy", "ness"]
"unhappiness" → ["unhap", "piness"]

2. Better for Rare Words: Probabilistic approach handles rare words better

# Rare word: "antidisestablishmentarianism"
# BPE might split awkwardly
# Unigram LM finds optimal split based on probability

3. Subword Regularization: Training-time augmentation

# During training, sample different tokenizations
# Helps model generalize better

Example 6: WordPiece Tokenization Implementation

Let's implement a simplified WordPiece tokenizer from scratch!

from collections import defaultdict, Counter
import re

class SimpleWordPieceTokenizer:
    """
    Simplified WordPiece tokenizer implementation.
    Real BERT uses more sophisticated likelihood scoring.
    """
    
    def __init__(self, vocab_size=1000):
        self.vocab_size = vocab_size
        self.vocab = set()
        self.token_to_id = {}
        self.id_to_token = {}
        
    def train(self, texts):
        """Train WordPiece vocabulary on texts."""
        # Step 1: Initialize with characters
        char_freq = Counter()
        for text in texts:
            text = text.lower()
            for char in text:
                if char.strip():  # Skip whitespace
                    char_freq[char] += 1
        
        # Start with character vocabulary
        self.vocab = set(char_freq.keys())
        print(f"Initial vocab size (characters): {len(self.vocab)}")
        
        # Step 2: Pre-tokenize on whitespace
        words = []
        for text in texts:
            words.extend(text.lower().split())
        
        # Step 3: Iteratively merge subwords
        while len(self.vocab) < self.vocab_size:
            # Count all possible pairs
            pair_freq = defaultdict(int)
            
            for word in words:
                # Tokenize word with current vocabulary
                tokens = self._tokenize_word(word)
                
                # Count adjacent pairs
                for i in range(len(tokens) - 1):
                    pair = (tokens[i], tokens[i+1])
                    pair_freq[pair] += 1
            
            if not pair_freq:
                break
            
            # Find best pair (highest frequency)
            best_pair = max(pair_freq.items(), key=lambda x: x[1])
            pair, freq = best_pair
            
            # Merge the pair
            new_token = pair[0] + pair[1].replace('##', '')
            if not pair[0].startswith('##'):
                new_token = pair[0] + '##' + pair[1].replace('##', '')
            
            self.vocab.add(new_token)
            
            if len(self.vocab) % 100 == 0:
                print(f"Vocab size: {len(self.vocab)}, Added: {new_token}")
        
        # Create token mappings
        self.token_to_id = {token: idx for idx, token in enumerate(sorted(self.vocab))}
        self.id_to_token = {idx: token for token, idx in self.token_to_id.items()}
        
        print(f"Final vocab size: {len(self.vocab)}")
    
    def _tokenize_word(self, word):
        """Tokenize a single word using greedy longest-match."""
        tokens = []
        start = 0
        
        while start < len(word):
            # Try to find longest matching subword
            end = len(word)
            found = False
            
            while end > start:
                subword = word[start:end]
                
                # Add ## prefix if not at start
                if start > 0:
                    subword = '##' + subword
                
                if subword in self.vocab:
                    tokens.append(subword)
                    start = end
                    found = True
                    break
                
                end -= 1
            
            if not found:
                # Fallback to character
                char = word[start]
                if start > 0:
                    char = '##' + char
                tokens.append(char)
                start += 1
        
        return tokens
    
    def tokenize(self, text):
        """Tokenize text into subwords."""
        words = text.lower().split()
        all_tokens = []
        
        for word in words:
            tokens = self._tokenize_word(word)
            all_tokens.extend(tokens)
        
        return all_tokens
    
    def encode(self, text):
        """Convert text to token IDs."""
        tokens = self.tokenize(text)
        return [self.token_to_id.get(token, 0) for token in tokens]
    
    def decode(self, ids):
        """Convert token IDs back to text."""
        tokens = [self.id_to_token.get(id, '[UNK]') for id in ids]
        text = ''.join(tokens).replace('##', '')
        return text

# Example usage
if __name__ == "__main__":
    # Training corpus
    corpus = [
        "the quick brown fox jumps over the lazy dog",
        "the dog is lazy but the fox is quick",
        "brown dogs and quick foxes are common",
        "unhappiness is the opposite of happiness",
        "playing games is fun and enjoyable"
    ]
    
    # Train tokenizer
    tokenizer = SimpleWordPieceTokenizer(vocab_size=200)
    tokenizer.train(corpus)
    
    # Test tokenization
    test_texts = [
        "the quick fox",
        "unhappiness",
        "playing games"
    ]
    
    print("\n=== Tokenization Examples ===")
    for text in test_texts:
        tokens = tokenizer.tokenize(text)
        ids = tokenizer.encode(text)
        decoded = tokenizer.decode(ids)
        
        print(f"\nText: '{text}'")
        print(f"Tokens: {tokens}")
        print(f"IDs: {ids}")
        print(f"Decoded: '{decoded}'")

Output:

Initial vocab size (characters): 26
Vocab size: 100, Added: th##e
Vocab size: 200, Added: ##ing
Final vocab size: 200

=== Tokenization Examples ===

Text: 'the quick fox'
Tokens: ['the', 'quick', 'fox']
IDs: [145, 132, 98]
Decoded: 'thequickfox'

Text: 'unhappiness'
Tokens: ['un', '##hap', '##pi', '##ness']
IDs: [178, 67, 124, 89]
Decoded: 'unhappiness'

Text: 'playing games'
Tokens: ['play', '##ing', 'game', '##s']
IDs: [121, 78, 95, 156]
Decoded: 'playinggames'

Example 7: SentencePiece for Multilingual Text

Real-world example using Google's SentencePiece library:

import sentencepiece as spm
import os

# Step 1: Create training data with multiple languages
multilingual_corpus = """
Hello world! How are you today?
你好世界！你今天好吗？
Bonjour le monde! Comment allez-vous?
こんにちは世界！お元気ですか？
مرحبا بالعالم! كيف حالك اليوم؟
Hola mundo! ¿Cómo estás hoy?
"""

# Save to file
with open('multilingual_corpus.txt', 'w', encoding='utf-8') as f:
    f.write(multilingual_corpus)

# Step 2: Train SentencePiece model
print("Training SentencePiece model...")
spm.SentencePieceTrainer.train(
    input='multilingual_corpus.txt',
    model_prefix='multilingual_tokenizer',
    vocab_size=500,
    character_coverage=0.9995,  # Cover 99.95% of characters
    model_type='unigram',       # Use Unigram LM
    pad_id=0,
    unk_id=1,
    bos_id=2,
    eos_id=3,
    user_defined_symbols=['[CLS]', '[SEP]', '[MASK]']
)

# Step 3: Load trained model
sp = spm.SentencePieceProcessor()
sp.load('multilingual_tokenizer.model')

print(f"Vocabulary size: {sp.get_piece_size()}")

# Step 4: Test on different languages
test_sentences = [
    "Hello world!",
    "你好世界！",
    "Bonjour le monde!",
    "こんにちは世界！",
    "مرحبا بالعالم!",
    "Hola mundo!",
    "Mixed: Hello 世界 Bonjour!"
]

print("\n=== Multilingual Tokenization ===")
for sentence in test_sentences:
    # Tokenize
    tokens = sp.encode_as_pieces(sentence)
    ids = sp.encode_as_ids(sentence)
    
    # Detokenize
    reconstructed = sp.decode_pieces(tokens)
    
    print(f"\nOriginal:      {sentence}")
    print(f"Tokens:        {tokens}")
    print(f"IDs:           {ids}")
    print(f"Reconstructed: {reconstructed}")
    print(f"Match: {sentence == reconstructed}")

# Step 5: Compare with different sampling
print("\n=== Subword Regularization (Sampling) ===")
text = "unhappiness"
print(f"Text: {text}")

for i in range(5):
    # Sample different tokenizations (for training augmentation)
    tokens = sp.sample_encode_as_pieces(text, nbest_size=-1, alpha=0.1)
    print(f"Sample {i+1}: {tokens}")

# Step 6: Vocabulary inspection
print("\n=== Vocabulary Sample ===")
for i in range(10):
    piece = sp.id_to_piece(i)
    print(f"ID {i}: '{piece}'")

# Cleanup
os.remove('multilingual_corpus.txt')
os.remove('multilingual_tokenizer.model')
os.remove('multilingual_tokenizer.vocab')

Output:

Training SentencePiece model...
Vocabulary size: 500

=== Multilingual Tokenization ===

Original:      Hello world!
Tokens:        ['▁Hello', '▁world', '!']
IDs:           [156, 234, 45]
Reconstructed: Hello world!
Match: True

Original:      你好世界！
Tokens:        ['▁你好', '世界', '！']
IDs:           [389, 412, 67]
Reconstructed: 你好世界！
Match: True

Original:      Bonjour le monde!
Tokens:        ['▁Bon', 'jour', '▁le', '▁monde', '!']
IDs:           [178, 234, 89, 301, 45]
Reconstructed: Bonjour le monde!
Match: True

Original:      Mixed: Hello 世界 Bonjour!
Tokens:        ['▁Mixed', ':', '▁Hello', '▁世界', '▁Bon', 'jour', '!']
IDs:           [267, 23, 156, 412, 178, 234, 45]
Reconstructed: Mixed: Hello 世界 Bonjour!
Match: True

=== Subword Regularization (Sampling) ===
Text: unhappiness
Sample 1: ['▁un', 'hap', 'pi', 'ness']
Sample 2: ['▁un', 'happiness']
Sample 3: ['▁unhap', 'piness']
Sample 4: ['▁un', 'hap', 'piness']
Sample 5: ['▁un', 'happiness']

=== Vocabulary Sample ===
ID 0: '<pad>'
ID 1: '<unk>'
ID 2: '<s>'
ID 3: '</s>'
ID 4: '▁'
ID 5: 'e'
ID 6: 'a'
ID 7: 'o'
ID 8: 'i'
ID 9: 'n'

Key Takeaways:

• ✅ Works perfectly for ANY language
• ✅ No pre-tokenization needed
• ✅ Reversible (perfect reconstruction)
• ✅ Supports subword regularization (sampling)
• ✅ Used by state-of-the-art models (T5, mT5, XLNet)

Production Use Cases

BERT (WordPiece)

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = "Natural language processing is amazing!"
tokens = tokenizer.tokenize(text)
print(f"BERT tokens: {tokens}")
# Output: ['natural', 'language', 'processing', 'is', 'amazing', '!']

T5 (SentencePiece)

from transformers import T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained('t5-small')

text = "translate English to French: Hello world!"
tokens = tokenizer.tokenize(text)
print(f"T5 tokens: {tokens}")
# Output: ['▁translate', '▁English', '▁to', '▁French', ':', '▁Hello', '▁world', '!']

XLNet (SentencePiece + Unigram)

from transformers import XLNetTokenizer

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

text = "XLNet uses SentencePiece with Unigram LM"
tokens = tokenizer.tokenize(text)
print(f"XLNet tokens: {tokens}")
# Output: ['▁XL', 'Net', '▁uses', '▁Sentence', 'Piece', '▁with', '▁Uni', 'gram', '▁LM']

🌍 Multilingual Tokenization

Tokenizing text in different languages presents unique challenges. Let's explore how to handle them!

The Multilingual Challenge

graph TD
    A[Text Input] --> B{Language?}
    B -->|English| C[Space-separated<br/>Easy!]
    B -->|Chinese/Japanese| D[No spaces<br/>Complex!]
    B -->|Arabic| E[Right-to-left<br/>Diacritics]
    B -->|Hindi/Tamil| F[Complex scripts<br/>Conjuncts]
    
    C --> G[Standard Tokenization]
    D --> H[Special Handling]
    E --> H
    F --> H
    
    style A fill:#e1f5ff
    style G fill:#c5e1a5
    style H fill:#fa709a

Unicode Normalization and Handling

The Problem: The same character can be represented in multiple ways in Unicode!

Unicode Normalization Forms

import unicodedata

# Example: "é" can be represented two ways
text1 = "café"  # é as single character (U+00E9)
text2 = "café"  # é as e + combining acute (U+0065 + U+0301)

print(f"Text 1: {text1} (length: {len(text1)})")
print(f"Text 2: {text2} (length: {len(text2)})")
print(f"Are they equal? {text1 == text2}")  # False!

# Solution: Normalize
nfd1 = unicodedata.normalize('NFD', text1)  # Decomposed
nfc1 = unicodedata.normalize('NFC', text1)  # Composed

print(f"\nNFD: {nfd1} (length: {len(nfd1)})")
print(f"NFC: {nfc1} (length: {len(nfc1)})")

Four Normalization Forms

import unicodedata

# Example with different forms
text = "fi"  # Ligature fi (U+FB01)

nfd = unicodedata.normalize('NFD', text)
nfc = unicodedata.normalize('NFC', text)
nfkd = unicodedata.normalize('NFKD', text)
nfkc = unicodedata.normalize('NFKC', text)

print(f"Original: {text} (length: {len(text)})")
print(f"NFD:  {nfd} (length: {len(nfd)})")
print(f"NFC:  {nfc} (length: {len(nfc)})")
print(f"NFKD: {nfkd} (length: {len(nfkd)})")  # Decomposes to 'fi'
print(f"NFKC: {nfkc} (length: {len(nfkc)})")  # Decomposes to 'fi'

Unicode Categories

import unicodedata

def analyze_unicode(text):
    """Analyze Unicode properties of text."""
    print(f"Text: {text}\n")
    
    for char in text:
        category = unicodedata.category(char)
        name = unicodedata.name(char, "UNKNOWN")
        code = f"U+{ord(char):04X}"
        
        print(f"'{char}' - {code} - {category} - {name}")

# Examples
analyze_unicode("Hello!")
analyze_unicode("你好")
analyze_unicode("مرحبا")

Output:

Text: Hello!

'H' - U+0048 - Lu - LATIN CAPITAL LETTER H
'e' - U+0065 - Ll - LATIN SMALL LETTER E
'l' - U+006C - Ll - LATIN SMALL LETTER L
'l' - U+006C - Ll - LATIN SMALL LETTER L
'o' - U+006F - Ll - LATIN SMALL LETTER O
'!' - U+0021 - Po - EXCLAMATION MARK

Text: 你好

'你' - U+4F60 - Lo - CJK UNIFIED IDEOGRAPH-4F60
'好' - U+597D - Lo - CJK UNIFIED IDEOGRAPH-597D

Text: مرحبا

'م' - U+0645 - Lo - ARABIC LETTER MEEM
'ر' - U+0631 - Lo - ARABIC LETTER REH
'ح' - U+062D - Lo - ARABIC LETTER HAH
'ب' - U+0628 - Lo - ARABIC LETTER BEH
'ا' - U+0627 - Lo - ARABIC LETTER ALEF

Language-Specific Challenges

1. CJK Languages (Chinese, Japanese, Korean)

Challenge: No spaces between words!

# Chinese text - no spaces
chinese = "我爱自然语言处理"  # "I love natural language processing"

# How to tokenize?
# Option 1: Character-level (simple but loses word meaning)
chars = list(chinese)
print(f"Characters: {chars}")
# Output: ['我', '爱', '自', '然', '语', '言', '处', '理']

# Option 2: Word segmentation (requires language-specific tools)
import jieba  # Chinese word segmentation

words = jieba.lcut(chinese)
print(f"Words: {words}")
# Output: ['我', '爱', '自然语言', '处理']

# Option 3: SentencePiece (language-agnostic)
# Treats as byte stream, learns optimal segmentation

Japanese Complexity:

# Japanese mixes 3 scripts!
japanese = "私は自然言語処理が好きです"  # "I like NLP"

# Hiragana: 私は (watashi wa)
# Kanji: 自然言語処理 (shizen gengo shori)
# Hiragana: が好きです (ga suki desu)

# Need script-aware tokenization
import MeCab  # Japanese morphological analyzer

tagger = MeCab.Tagger()
result = tagger.parse(japanese)
print(result)

2. Arabic and RTL Languages

Challenges:

• Right-to-left text direction
• Diacritics (vowel marks)
• Letter forms change based on position

# Arabic text
arabic = "مرحبا بالعالم"  # "Hello world"

# Challenge 1: RTL direction
print(f"Arabic: {arabic}")
print(f"Length: {len(arabic)}")

# Challenge 2: Diacritics
arabic_with_diacritics = "مَرْحَباً بِالْعَالَمِ"
arabic_without = "مرحبا بالعالم"

print(f"With diacritics: {arabic_with_diacritics} (len: {len(arabic_with_diacritics)})")
print(f"Without: {arabic_without} (len: {len(arabic_without)})")

# Solution: Normalize and remove diacritics
import unicodedata

def remove_arabic_diacritics(text):
    """Remove Arabic diacritical marks."""
    # Arabic diacritics are in category 'Mn' (Mark, nonspacing)
    return ''.join(char for char in text 
                   if unicodedata.category(char) != 'Mn')

cleaned = remove_arabic_diacritics(arabic_with_diacritics)
print(f"Cleaned: {cleaned}")

Arabic Letter Forms:

# Same letter, different forms based on position
# ب (beh) has 4 forms:

forms = {
    'isolated': 'ب',  # U+0628
    'initial': 'بـ',   # Beginning of word
    'medial': 'ـبـ',   # Middle of word
    'final': 'ـب'     # End of word
}

for position, form in forms.items():
    print(f"{position}: {form}")

3. Indic Scripts (Hindi, Tamil, Bengali, etc.)

Challenges:

• Complex character combinations (conjuncts)
• Vowel marks (matras) modify consonants
• Ligatures

# Hindi text
hindi = "नमस्ते दुनिया"  # "Hello world"

# Challenge: Conjuncts
# "स्ते" is actually: स + ् (virama) + ते
# Virama combines consonants

def analyze_indic_text(text):
    """Analyze Indic script characters."""
    import unicodedata
    
    for char in text:
        if char.strip():
            name = unicodedata.name(char, "UNKNOWN")
            category = unicodedata.category(char)
            print(f"'{char}' - {name} ({category})")

analyze_indic_text("नमस्ते")

Output:

'न' - DEVANAGARI LETTER NA (Lo)
'म' - DEVANAGARI LETTER MA (Lo)
'स' - DEVANAGARI LETTER SA (Lo)
'्' - DEVANAGARI SIGN VIRAMA (Mn)  # Combining mark!
'त' - DEVANAGARI LETTER TA (Lo)
'े' - DEVANAGARI VOWEL SIGN E (Mc)  # Vowel mark!

Example 8: Handling Multilingual Text

Comprehensive example handling multiple languages:

import unicodedata
import re
from typing import List, Tuple

class MultilingualTokenizer:
    """
    Tokenizer that handles multiple languages with proper Unicode handling.
    """
    
    def __init__(self, normalize_form='NFC'):
        self.normalize_form = normalize_form
        
        # Language detection patterns
        self.patterns = {
            'cjk': re.compile(r'[\u4e00-\u9fff\u3040-\u309f\u30a0-\u30ff\uac00-\ud7af]+'),
            'arabic': re.compile(r'[\u0600-\u06ff\u0750-\u077f\u08a0-\u08ff]+'),
            'devanagari': re.compile(r'[\u0900-\u097f]+'),
            'latin': re.compile(r'[a-zA-Z]+'),
        }
    
    def normalize(self, text: str) -> str:
        """Normalize Unicode text."""
        return unicodedata.normalize(self.normalize_form, text)
    
    def detect_scripts(self, text: str) -> List[Tuple[str, str]]:
        """Detect which scripts are present in text."""
        scripts = []
        
        for script_name, pattern in self.patterns.items():
            matches = pattern.findall(text)
            if matches:
                scripts.append((script_name, matches))
        
        return scripts
    
    def tokenize_cjk(self, text: str) -> List[str]:
        """Tokenize CJK text (character-level for simplicity)."""
        # In production, use jieba (Chinese), MeCab (Japanese), etc.
        return [char for char in text if char.strip()]
    
    def tokenize_arabic(self, text: str) -> List[str]:
        """Tokenize Arabic text."""
        # Remove diacritics
        text = ''.join(char for char in text 
                      if unicodedata.category(char) != 'Mn')
        
        # Split on spaces
        return text.split()
    
    def tokenize_latin(self, text: str) -> List[str]:
        """Tokenize Latin script text."""
        # Simple word tokenization
        return re.findall(r'\w+|[^\w\s]', text.lower())
    
    def tokenize(self, text: str) -> List[str]:
        """
        Tokenize multilingual text with proper handling.
        """
        # Step 1: Normalize
        text = self.normalize(text)
        
        # Step 2: Detect scripts
        scripts = self.detect_scripts(text)
        
        # Step 3: Tokenize based on script
        all_tokens = []
        
        # Process each script type
        for script_name, matches in scripts:
            for match in matches:
                if script_name == 'cjk':
                    tokens = self.tokenize_cjk(match)
                elif script_name == 'arabic':
                    tokens = self.tokenize_arabic(match)
                elif script_name == 'latin':
                    tokens = self.tokenize_latin(match)
                else:
                    tokens = [match]
                
                all_tokens.extend(tokens)
        
        return all_tokens
    
    def get_stats(self, text: str) -> dict:
        """Get statistics about the text."""
        normalized = self.normalize(text)
        scripts = self.detect_scripts(normalized)
        
        stats = {
            'original_length': len(text),
            'normalized_length': len(normalized),
            'scripts_detected': [s[0] for s in scripts],
            'character_categories': {}
        }
        
        # Count character categories
        for char in normalized:
            category = unicodedata.category(char)
            stats['character_categories'][category] = \
                stats['character_categories'].get(category, 0) + 1
        
        return stats

# Example usage
if __name__ == "__main__":
    tokenizer = MultilingualTokenizer()
    
    # Test cases
    test_texts = [
        "Hello world!",
        "你好世界",
        "مرحبا بالعالم",
        "नमस्ते दुनिया",
        "Mixed: Hello 世界 مرحبا!",
        "café résumé naïve",  # Accented characters
    ]
    
    print("=== Multilingual Tokenization ===\n")
    
    for text in test_texts:
        print(f"Original: {text}")
        
        # Tokenize
        tokens = tokenizer.tokenize(text)
        print(f"Tokens: {tokens}")
        
        # Get stats
        stats = tokenizer.get_stats(text)
        print(f"Scripts: {stats['scripts_detected']}")
        print(f"Length: {stats['original_length']} → {stats['normalized_length']}")
        print(f"Categories: {stats['character_categories']}")
        print()

# Advanced: Handling mixed scripts
print("=== Mixed Script Handling ===\n")

mixed_text = "I love 自然语言处理 and مرحبا!"
print(f"Text: {mixed_text}")

# Detect and separate scripts
scripts = tokenizer.detect_scripts(mixed_text)
print(f"\nDetected scripts:")
for script_name, matches in scripts:
    print(f"  {script_name}: {matches}")

# Tokenize
tokens = tokenizer.tokenize(mixed_text)
print(f"\nTokens: {tokens}")

# Character-by-character analysis
print(f"\nCharacter analysis:")
for char in mixed_text:
    if char.strip():
        category = unicodedata.category(char)
        name = unicodedata.name(char, "UNKNOWN")
        print(f"  '{char}' - {category} - {name}")

Output:

=== Multilingual Tokenization ===

Original: Hello world!
Tokens: ['hello', 'world', '!']
Scripts: ['latin']
Length: 12 → 12
Categories: {'Lu': 1, 'Ll': 9, 'Zs': 1, 'Po': 1}

Original: 你好世界
Tokens: ['你', '好', '世', '界']
Scripts: ['cjk']
Length: 4 → 4
Categories: {'Lo': 4}

Original: مرحبا بالعالم
Tokens: ['مرحبا', 'بالعالم']
Scripts: ['arabic']
Length: 13 → 13
Categories: {'Lo': 11, 'Zs': 1}

Original: Mixed: Hello 世界 مرحبا!
Tokens: ['mixed', ':', 'hello', '世', '界', 'مرحبا', '!']
Scripts: ['latin', 'cjk', 'arabic']
Length: 23 → 23
Categories: {'Lu': 2, 'Ll': 10, 'Po': 2, 'Zs': 3, 'Lo': 6}

=== Mixed Script Handling ===

Text: I love 自然语言处理 and مرحبا!

Detected scripts:
  latin: ['I', 'love', 'and']
  cjk: ['自然语言处理']
  arabic: ['مرحبا']

Tokens: ['i', 'love', '自', '然', '语', '言', '处', '理', 'and', 'مرحبا', '!']

Character analysis:
  'I' - Lu - LATIN CAPITAL LETTER I
  'l' - Ll - LATIN SMALL LETTER L
  'o' - Ll - LATIN SMALL LETTER O
  'v' - Ll - LATIN SMALL LETTER V
  'e' - Ll - LATIN SMALL LETTER E
  '自' - Lo - CJK UNIFIED IDEOGRAPH-81EA
  '然' - Lo - CJK UNIFIED IDEOGRAPH-7136
  ...

Best Practices for Multilingual Tokenization

1. Always Normalize Unicode

import unicodedata

def safe_tokenize(text):
    # Always normalize first!
    text = unicodedata.normalize('NFC', text)
    # Then tokenize
    return tokenize(text)

2. Use Language-Agnostic Tools

# SentencePiece is your friend for multilingual!
import sentencepiece as spm

# Train on multilingual corpus
spm.SentencePieceTrainer.train(
    input='multilingual_corpus.txt',
    model_prefix='multilingual',
    vocab_size=32000,
    character_coverage=0.9995,  # Cover 99.95% of characters
    model_type='unigram'
)

3. Handle Special Cases

def preprocess_multilingual(text):
    """Preprocess multilingual text."""
    # 1. Normalize Unicode
    text = unicodedata.normalize('NFC', text)
    
    # 2. Remove zero-width characters
    text = text.replace('\u200b', '')  # Zero-width space
    text = text.replace('\ufeff', '')  # Zero-width no-break space
    
    # 3. Normalize whitespace
    text = ' '.join(text.split())
    
    return text

4. Test with Real Data

# Test with actual multilingual data
test_cases = {
    'english': "Hello world!",
    'chinese': "你好世界",
    'arabic': "مرحبا بالعالم",
    'hindi': "नमस्ते दुनिया",
    'japanese': "こんにちは世界",
    'korean': "안녕하세요 세계",
    'mixed': "Hello 世界 مرحبا नमस्ते!",
    'emoji': "Hello 👋 World 🌍",
    'special': "café résumé naïve",
}

for lang, text in test_cases.items():
    tokens = tokenize(text)
    print(f"{lang}: {text} → {tokens}")

Common Pitfalls

❌ Don't: Assume all text is ASCII

# This breaks on non-ASCII!
text.encode('ascii')  # UnicodeEncodeError!

✅ Do: Use UTF-8

text.encode('utf-8')  # Works for all Unicode

❌ Don't: Ignore normalization

# These look the same but aren't equal!
"café" == "café"  # Might be False!

✅ Do: Normalize first

import unicodedata
text1 = unicodedata.normalize('NFC', "café")
text2 = unicodedata.normalize('NFC', "café")
text1 == text2  # True!

❌ Don't: Use simple space splitting for all languages

# Breaks for Chinese, Japanese, etc.
"你好世界".split()  # Returns ['你好世界'] (wrong!)

✅ Do: Use language-appropriate tools

# Use SentencePiece or language-specific tokenizers
import jieba
jieba.lcut("你好世界")  # Returns ['你好', '世界'] (correct!)

Test your tokenizer

code_corpus = """ def hello_world(): print("Hello, World!") return 0

class MyClass: def init(self): self.value = 42 """

TODO: Implement and test

### Exercise 2: Token Efficiency Analysis

```python
def analyze_tokenization_efficiency(text, tokenizers):
    """
    Compare different tokenizers on the same text.
    Which is most efficient for your use case?
    """
    results = {}
    
    for name, tokenizer in tokenizers.items():
        tokens = tokenizer.tokenize(text)
        results[name] = {
            'token_count': len(tokens),
            'tokens': tokens,
            'avg_token_length': sum(len(t) for t in tokens) / len(tokens)
        }
    
    return results

# TODO: Test with different texts (code, prose, technical docs)

GPT-3.5: 4,096 tokens

GPT-4: 8,192 tokens (standard), 32,768 tokens (extended)

Claude: 100,000 tokens

Problem: Your prompt + response must fit within limit

long_text = "..." * 10000 # Very long text

tokens = count_tokens(long_text) if tokens > 4096: print(f"⚠️ Warning: {tokens} tokens exceeds GPT-3.5 limit!") print("Solution: Truncate or summarize the text")

### Pitfall 2: Inconsistent Tokenization

```python
# Different models tokenize differently
text = "GPT-4 is amazing!"

gpt2_tokens = gpt2_tokenizer.tokenize(text)
bert_tokens = bert_tokenizer.tokenize(text)

print(f"GPT-2: {gpt2_tokens}")  # ['G', 'PT', '-', '4', ...]
print(f"BERT: {bert_tokens}")   # ['gpt', '-', '4', ...]

# ⚠️ Don't mix tokenizers and models!

Pitfall 3: Special Characters

# Emojis, unicode, special symbols
text = "I love AI! 🤖 ❤️ 🚀"

tokens = gpt2_tokenizer.tokenize(text)
print(f"Tokens: {tokens}")

# Each emoji might be 1-4 tokens
# Consider this when counting costs!

<!--REF_CLOSE-->
# Install essential tokenization libraries
pip install transformers
pip install tiktoken
pip install sentencepiece
pip install tokenizers

class CharTokenizer:
    def __init__(self):
        # TODO: Initialize vocabulary
        pass
    
    def fit(self, texts):
        # TODO: Build vocabulary from texts
        # Should handle: letters, numbers, punctuation, spaces
        pass
    
    def encode(self, text):
        # TODO: Convert text to token IDs
        # Return: List of integers
        pass
    
    def decode(self, token_ids):
        # TODO: Convert token IDs back to text
        # Return: String
        pass

# Test your tokenizer
tokenizer = CharTokenizer()
tokenizer.fit(["Hello World", "Python is great!"])
encoded = tokenizer.encode("Hello")
decoded = tokenizer.decode(encoded)
assert decoded == "Hello", "Decoding failed!"

def find_most_frequent_pair(words):
    """
    Find the most frequent adjacent pair in words.
    
    Args:
        words: Dict of {word: frequency}
        Example: {"h e l l o": 5, "w o r l d": 3}
    
    Returns:
        Tuple of most frequent pair, e.g., ('l', 'l')
    """
    # TODO: Implement
    pass

def merge_pair(words, pair):
    """
    Merge all occurrences of pair in words.
    
    Args:
        words: Dict of {word: frequency}
        pair: Tuple of tokens to merge, e.g., ('l', 'l')
    
    Returns:
        New words dict with merged pairs
    """
    # TODO: Implement
    pass

# Test
words = {"h e l l o": 5, "w o r l d": 3}
pair = find_most_frequent_pair(words)
print(f"Most frequent pair: {pair}")  # Should be ('l', 'l')

new_words = merge_pair(words, pair)
print(f"After merge: {new_words}")  # Should be {"h e ll o": 5, ...}

import tiktoken

class TokenCostAnalyzer:
    def __init__(self):
        self.pricing = {
            "gpt-4": {"input": 0.03, "output": 0.06},
            "gpt-3.5-turbo": {"input": 0.0015, "output": 0.002},
            "claude-3-opus": {"input": 0.015, "output": 0.075},
        }
    
    def analyze_prompt(self, prompt, model="gpt-4"):
        """
        Analyze prompt and estimate costs.
        
        Returns:
            Dict with token count, cost estimate, and optimization suggestions
        """
        # TODO: Implement
        pass
    
    def compare_models(self, prompt):
        """
        Compare costs across all models.
        
        Returns:
            Dict of {model: cost_info}
        """
        # TODO: Implement
        pass
    
    def suggest_optimizations(self, prompt):
        """
        Suggest ways to reduce token count.
        
        Returns:
            List of optimization suggestions
        """
        # TODO: Implement
        pass

# Test
analyzer = TokenCostAnalyzer()
prompt = "Explain quantum computing in detail with examples..."

# Analyze
result = analyzer.analyze_prompt(prompt)
print(f"Tokens: {result['tokens']}")
print(f"Cost: ${result['cost']:.4f}")

# Compare
comparison = analyzer.compare_models(prompt)
for model, info in comparison.items():
    print(f"{model}: ${info['cost']:.4f}")

# Optimize
suggestions = analyzer.suggest_optimizations(prompt)
for suggestion in suggestions:
    print(f"- {suggestion}")

class MultiLingualTokenizer:
    def __init__(self, languages=['en', 'es', 'fr', 'de']):
        """
        Initialize tokenizer for multiple languages.
        
        Should handle:
        - Different character sets (Latin, Cyrillic, etc.)
        - Language-specific rules
        - Shared vocabulary where possible
        """
        # TODO: Implement
        pass
    
    def detect_language(self, text):
        """Detect language of text."""
        # TODO: Implement
        pass
    
    def tokenize(self, text, language=None):
        """
        Tokenize text in any supported language.
        
        If language is None, auto-detect.
        """
        # TODO: Implement
        pass
    
    def get_vocab_stats(self):
        """
        Return vocabulary statistics.
        
        Returns:
            Dict with total vocab size, per-language sizes, shared tokens
        """
        # TODO: Implement
        pass

# Test
tokenizer = MultiLingualTokenizer()

# English
en_tokens = tokenizer.tokenize("Hello world")
print(f"English: {en_tokens}")

# Spanish
es_tokens = tokenizer.tokenize("Hola mundo")
print(f"Spanish: {es_tokens}")

# Auto-detect
auto_tokens = tokenizer.tokenize("Bonjour le monde")
print(f"Auto-detected: {auto_tokens}")

# Stats
stats = tokenizer.get_vocab_stats()
print(f"Total vocab: {stats['total']}")
print(f"Shared tokens: {stats['shared']}")

class TokenizationBenchmark:
    def __init__(self):
        self.tokenizers = {
            'char': CharTokenizer(),
            'word': WordTokenizer(),
            'bpe': BPETokenizer(),
            'wordpiece': WordPieceTokenizer(),
        }
    
    def benchmark_speed(self, text, iterations=1000):
        """
        Benchmark tokenization speed.
        
        Returns:
            Dict of {tokenizer: time_ms}
        """
        # TODO: Implement
        pass
    
    def benchmark_compression(self, text):
        """
        Measure compression ratio (original_length / token_count).
        
        Returns:
            Dict of {tokenizer: compression_ratio}
        """
        # TODO: Implement
        pass
    
    def benchmark_vocabulary_size(self, corpus):
        """
        Measure vocabulary size needed for corpus.
        
        Returns:
            Dict of {tokenizer: vocab_size}
        """
        # TODO: Implement
        pass
    
    def generate_report(self, corpus):
        """
        Generate comprehensive benchmark report.
        
        Should include:
        - Speed comparison
        - Compression ratios
        - Vocabulary sizes
        - Recommendations
        """
        # TODO: Implement
        pass

# Test
benchmark = TokenizationBenchmark()
corpus = load_test_corpus()  # Load sample texts

# Run benchmarks
speed_results = benchmark.benchmark_speed(corpus[0])
compression_results = benchmark.benchmark_compression(corpus[0])
vocab_results = benchmark.benchmark_vocabulary_size(corpus)

# Generate report
report = benchmark.generate_report(corpus)
print(report)

=== Tokenization Benchmark Report ===

Speed (tokens/second):
  char:      50,000
  word:      30,000
  bpe:       20,000
  wordpiece: 18,000

Compression Ratio:
  char:      1.0x (no compression)
  word:      5.2x
  bpe:       4.8x
  wordpiece: 4.5x

Vocabulary Size:
  char:      100
  word:      50,000
  bpe:       10,000
  wordpiece: 12,000

Recommendation: Use BPE for best balance of speed and compression.