🔤 WordPiece - BERT's Tokenization Algorithm

Used by: BERT, DistilBERT, ELECTRA, and most Google models

🎯 Why WordPiece?

The Problem with Pure Frequency

BPE: Merges most frequent pairs
Problem: "aa" might be more frequent than "er", but "er" is more meaningful

WordPiece: Merges pairs that maximize likelihood
Solution: Considers how well the merge explains the data

WordPiece Advantages

1. Likelihood-based - Smarter than frequency counting
2. Linguistically aware - Tends to create meaningful subwords
3. Proven at scale - Powers BERT and derivatives
4. Efficient - Similar performance to BPE

🔬 How WordPiece Works

Core Algorithm

1. Start with character vocabulary
2. For each possible merge:
   - Calculate: likelihood_increase = P(merged) / (P(left) * P(right))
3. Choose merge with highest likelihood increase
4. Repeat until desired vocabulary size

Likelihood Calculation

# BPE scoring (frequency-based)
score_bpe = count(pair)

# WordPiece scoring (likelihood-based)
score_wp = count(pair) / (count(left) * count(right))

Visual Example

Corpus: "the the the there"

BPE would merge:
('t','h') → 'th' (frequency: 4)

WordPiece considers:
('t','h'): count=4, left=4, right=4 → score = 4/(4*4) = 0.25
('h','e'): count=4, left=4, right=4 → score = 4/(4*4) = 0.25
('e','r'): count=1, left=4, right=1 → score = 1/(4*1) = 0.25

All equal! But WordPiece also considers context...

💻 WordPiece Implementation

Simplified WordPiece Tokenizer

from collections import Counter
import math

class SimpleWordPieceTokenizer:
    """
    Simplified WordPiece tokenizer (BERT-style).
    Uses likelihood-based scoring instead of frequency.
    """
    
    def __init__(self, vocab_size=1000):
        self.vocab_size = vocab_size
        self.vocab = set()
        self.merges = []
        
    def get_pair_scores(self, words):
        """Calculate likelihood-based scores for all pairs."""
        # Count individual tokens
        token_counts = Counter()
        for word, freq in words.items():
            for token in word.split():
                token_counts[token] += freq
        
        # Count pairs and calculate scores
        pair_scores = {}
        for word, freq in words.items():
            symbols = word.split()
            for i in range(len(symbols) - 1):
                pair = (symbols[i], symbols[i + 1])
                # Likelihood score: P(pair) / (P(left) * P(right))
                pair_count = freq
                left_count = token_counts[symbols[i]]
                right_count = token_counts[symbols[i + 1]]
                
                score = pair_count / (left_count * right_count)
                pair_scores[pair] = pair_scores.get(pair, 0) + score
        
        return pair_scores
    
    def merge_vocab(self, pair, words):
        """Merge the best pair in vocabulary."""
        new_words = {}
        bigram = ' '.join(pair)
        replacement = ''.join(pair)
        
        for word in words:
            new_word = word.replace(bigram, replacement)
            new_words[new_word] = words[word]
        return new_words
    
    def train(self, text):
        """Train WordPiece on text corpus."""
        # Start with character-level tokens
        words = {}
        for word in text.split():
            # Add ## prefix for subwords (BERT convention)
            word_tokens = ' '.join(['##' + c if i > 0 else c 
                                   for i, c in enumerate(word)]) + ' </w>'
            words[word_tokens] = words.get(word_tokens, 0) + 1
        
        # Iteratively merge best pairs
        for i in range(self.vocab_size - 256):
            pair_scores = self.get_pair_scores(words)
            if not pair_scores:
                break
            
            # Get pair with highest likelihood score
            best_pair = max(pair_scores, key=pair_scores.get)
            words = self.merge_vocab(best_pair, words)
            self.merges.append(best_pair)
            
            if i % 100 == 0:
                print(f"Merge {i}: {best_pair} (score: {pair_scores[best_pair]:.4f})")
        
        # Build vocabulary
        self.vocab = set()
        for word in words:
            self.vocab.update(word.split())
        
        print(f"\nTraining complete!")
        print(f"Vocabulary size: {len(self.vocab)}")
        print(f"Number of merges: {len(self.merges)}")
    
    def tokenize(self, text):
        """Tokenize text using learned WordPiece merges."""
        tokens = []
        for word in text.split():
            # Start with characters (BERT-style with ##)
            word_tokens = [word[0]] + ['##' + c for c in word[1:]] + ['</w>']
            
            # Apply merges in order
            for merge_pair in self.merges:
                i = 0
                new_tokens = []
                while i < len(word_tokens):
                    if i < len(word_tokens) - 1 and \
                       (word_tokens[i], word_tokens[i + 1]) == merge_pair:
                        new_tokens.append(word_tokens[i] + word_tokens[i + 1])
                        i += 2
                    else:
                        new_tokens.append(word_tokens[i])
                        i += 1
                word_tokens = new_tokens
            
            tokens.extend(word_tokens)
        
        return tokens

# Train the tokenizer
corpus = """
The quick brown fox jumps over the lazy dog.
The dog was not amused by the fox's antics.
"""

tokenizer = SimpleWordPieceTokenizer(vocab_size=300)
tokenizer.train(corpus)

# Test tokenization
test_text = "The quick fox"
tokens = tokenizer.tokenize(test_text)
print(f"\nTokenizing: '{test_text}'")
print(f"Tokens: {tokens}")

🎯 WordPiece in Production (BERT)

Using BERT's WordPiece Tokenizer

from transformers import AutoTokenizer

# Load BERT's WordPiece tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "Hello, how are you doing today?"

# Tokenize
tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print(f"Text: {text}")
print(f"Tokens: {tokens}")
print(f"Token IDs: {token_ids}")
print(f"Token count: {len(tokens)}")

# Decode back
decoded = tokenizer.decode(token_ids)
print(f"Decoded: {decoded}")

Output:

Text: Hello, how are you doing today?
Tokens: ['hello', ',', 'how', 'are', 'you', 'doing', 'today', '?']
Token IDs: [101, 7592, 1010, 2129, 2024, 2017, 2725, 2651, 1029, 102]
Token count: 8
Decoded: [CLS] hello, how are you doing today? [SEP]

Note: BERT adds special tokens [CLS] and [SEP] automatically

WordPiece Special Tokens

# BERT's special tokens
print(tokenizer.special_tokens_map)

Output:

{
    'unk_token': '[UNK]',      # Unknown token
    'sep_token': '[SEP]',      # Separator
    'pad_token': '[PAD]',      # Padding
    'cls_token': '[CLS]',      # Classification
    'mask_token': '[MASK]'     # Masked language modeling
}

🔍 WordPiece vs BPE

Key Differences

Example Comparison

from transformers import AutoTokenizer

gpt2_tokenizer = AutoTokenizer.from_pretrained("gpt2")
bert_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "unhappiness"

print("GPT-2 (BPE):", gpt2_tokenizer.tokenize(text))
print("BERT (WordPiece):", bert_tokenizer.tokenize(text))

Output:

GPT-2 (BPE): ['un', 'happiness']
BERT (WordPiece): ['un', '##hap', '##pi', '##ness']

🎯 WordPiece Characteristics

Strengths

✅ Linguistically aware - Creates meaningful subwords ✅ Likelihood-based - Smarter than frequency ✅ Proven at scale - Powers BERT family ✅ Rich special tokens - Good for NLU tasks

Weaknesses

❌ More complex - Harder to implement than BPE ❌ Slower training - Likelihood calculation overhead ❌ Less popular - BPE dominates in new models ❌ BERT-specific - Tied to BERT architecture

🎯 When to Use WordPiece

Use WordPiece when:

• Building BERT-style models
• Need linguistic awareness
• Working on NLU tasks (classification, NER, QA)
• Want rich special token support

Use BPE instead when:

• Building GPT-style models
• Need simplicity and speed
• Working on generation tasks
• Want wider community support

🎯 Practice Exercise

Challenge: Compare BPE and WordPiece on the same corpus:

1. Train both tokenizers on identical data
2. Compare vocabulary overlap
3. Measure tokenization efficiency
4. Analyze linguistic quality of subwords

Next: 01.c SentencePiece - Learn language-agnostic tokenization