WordPiece

WordPiece is a subword tokenization approach. Instead of requiring every complete word to exist in the vocabulary, it learns reusable pieces that can be combined to represent many words — including rare and unseen ones.

BPE merges the pair of symbols with the highest raw count. This can favour common pairs that add little linguistic value when combined. WordPiece asks a sharper question: does this merge increase the probability of the training corpus?

• Frequency vs Likelihood: BPE selects by raw co-occurrence count. WordPiece selects by mutual likelihood score, which normalises by how often each piece appears independently.
• Scoring formula: WordPiece selects the candidate merge pair $(u, v)$ that maximises $\frac{\text{count}(uv)}{\text{count}(u) \times \text{count}(v)}$ — pairs that appear together far more often than chance.
• Continuation marker: WordPiece uses ## prefixes (e.g. ##ing) to mark tokens that continue a previous word, cleanly distinguishing roots from bound morphemes.

Think of the WordPiece vocabulary as a set of puzzle pieces. A tokenizer picks the fewest, largest pieces that exactly cover each word. Pieces that start a word are bare (e.g. play). Pieces that continue a word carry the ## marker (e.g. ##ing).

• Common words may remain whole tokens in the vocabulary.
• Rare words are split into known pieces, preserving morphological information.
• The ## marker means "this piece continues the current word" — it is not a separator, it is a continuation flag.
• The tokenizer does not cut at arbitrary character positions; it selects from a learned vocabulary using a greedy longest-match strategy.

Illustrative example (toy vocabulary)

Vocabulary: ["play", "##ing", "##er", "player"]

playing → play  +  ##ing     (2 tokens)
player  → player              (1 token — whole word is in vocab)
played  → play  +  [UNK]     (##ed not in this toy vocab → [UNK] word)

A. Vocabulary learning

• Start with a corpus and initialise vocabulary with all unique base characters and special tokens ([PAD], [UNK], [CLS], [SEP], [MASK]).
• Train a unigram language model over the current corpus segmentation.
• Evaluate all adjacent pair merges; select the pair yielding the largest gain in corpus likelihood.
• Add the merged piece to vocabulary and repeat until reaching the target size (e.g. 30,522 for BERT).

B. Tokenization of new text (MaxMatch / Greedy)

• Pre-tokenize on whitespace and punctuation (split into words).
• For each word, find the longest substring starting at index 0 that exists in the vocabulary.
• For remaining characters, search for the longest matching substring prefixed with ##.
• If any segment cannot be matched, the entire word becomes [UNK].
• Convert selected pieces to token IDs.

Unlike BPE, which replays an ordered list of merge rules iteratively, WordPiece tokenization at inference runs a greedy longest-matching prefix algorithm (MaxMatch) per word:

WordPiece inference loop (per word):

start = 0
tokens = []

while start < len(word):
    end = len(word)
    cur_substr = None
    while start < end:
        substr = word[start:end]
        if start > 0:
            substr = "##" + substr
        if substr in vocabulary:
            cur_substr = substr
            break
        end -= 1

    if cur_substr is None:
        return ["[UNK]"]       # whole word is unknown

    tokens.append(cur_substr)
    start = end

return tokens

Example: tokenizing "playing"

Use a tiny toy vocabulary to understand the mechanics. This is a learning exercise, not the full production algorithm.

Now the failure case — what if ##happi is not in the vocabulary?

Input: "unhappily"
Toy vocab: {"un", "##happy", "##ness", "happy", "##ly"}

Step 1: "un"  → found. Token = "un", advance to index 2.
Step 2: remaining = "happily"
  Try "##happily" → No
  Try "##happil"  → No
  Try "##happi"   → No
  Try "##happ"    → No
  Try "##hap"     → No
  Try "##ha"      → No
  Try "##h"       → No  ← no match at all

Result: entire word → ["[UNK]"]

Toy vocabulary table used by the demo below:

Type a word below and see the greedy segmentation. Words outside the toy vocabulary (e.g. running) will return [UNK].

A minimal, readable implementation of the WordPiece MaxMatch tokenization loop. Focus is on the vocabulary lookup and ## prefix logic — the same pattern used in production implementations like HuggingFace Tokenizers.

def wordpiece_tokenize(word: str, vocab: set, unk_token: str = "[UNK]") -> list[str]:
    """
    Greedy longest-match (MaxMatch) WordPiece tokenization for a single word.

    Args:
        word:      The word to tokenize (already whitespace-split).
        vocab:     Set of known vocabulary tokens (including ## continuation pieces).
        unk_token: Token to use when no segmentation is possible.

    Returns:
        List of subword tokens, or [unk_token] if any segment is unresolvable.
    """
    tokens = []
    start = 0
    while start < len(word):
        end = len(word)
        matched_token = None
        while start < end:
            substr = word[start:end]
            if start > 0:
                substr = "##" + substr      # continuation marker
            if substr in vocab:
                matched_token = substr
                break
            end -= 1

        if matched_token is None:
            return [unk_token]              # whole word is unknown

        tokens.append(matched_token)
        start = end

    return tokens

# ── Toy vocabulary ──────────────────────────────────────────────────────────
vocab = {"play", "player", "##ing", "##er", "##s", "##ed", "##ly"}

print(wordpiece_tokenize("playing", vocab))   # ['play', '##ing']
print(wordpiece_tokenize("played",  vocab))   # ['play', '##ed']
print(wordpiece_tokenize("players", vocab))   # ['player', '##s']
print(wordpiece_tokenize("playly",  vocab))   # ['play', '##ly']
print(wordpiece_tokenize("unknown", vocab))   # ['[UNK]']

# ── Compare with a real BERT tokenizer ──────────────────────────────────────
from transformers import BertTokenizer
bert = BertTokenizer.from_pretrained("bert-base-uncased")
print(bert.tokenize("unconditionally"))
# ['un', '##condit', '##ionally']  (actual BERT output may vary)

• BERT family: BERT-base, BERT-large, DistilBERT, and related models use WordPiece with a 30,522-token vocabulary built on English Wikipedia + BookCorpus.
• Sequence length & truncation: Every subword segment is a token. A single long or technical word can become 4–6 tokens, quickly consuming the 512-token limit. Understanding this prevents silent truncation bugs.
• ## in production: When post-processing BERT outputs (e.g. NER, span extraction), ## tokens must be merged back into words. Ignoring them produces fragmented predictions.
• Case sensitivity: bert-base-uncased lowercases all text before tokenization; bert-base-cased preserves case. Using the wrong one changes vocabulary lookups and model accuracy.

• Different tokenizers split the same word differently depending on training corpus, vocabulary size, and algorithm. Never assume cross-tokenizer compatibility.
• Token efficiency is language-dependent — a vocabulary built on English performs poorly on morphologically rich or logographic languages without dedicated training.
• WordPiece reduces the unknown-word problem but does not guarantee semantically ideal segmentations (e.g. transformers → transform + ##ers loses the proper-noun meaning).
• Traditional WordPiece can still emit [UNK] if a base character is missing from the initial vocabulary.
• ## is not universal: Only BERT-style WordPiece uses ##. BPE uses Ġ or _; SentencePiece uses ▁. Mixing tokenizers causes silent data corruption.

Common misconceptions to tackle:

• "WordPiece and BPE are identical" — No: different selection criteria, different inference algorithms.
• "The ## means a new word starts" — No: ## means the piece continues the current word.
• "WordPiece will always find a valid segmentation" — No: if any segment fails, the entire word becomes [UNK].
• "I can swap BERT's tokenizer with GPT-2's tokenizer" — No: different vocabularies, different markers, different segmentations.

Summary

• Problem: Word-level vocabularies are inefficient and struggle with rare words; character-level sequences are too long.
• Idea: Represent text using reusable subword pieces, marking continuation pieces with ##.
• Vocabulary selection: WordPiece scores candidate merges by corpus likelihood — not raw frequency.
• Tokenization: Greedy longest-matching (MaxMatch) per word against the learned vocabulary set.
• vs BPE: Both are subword methods, but BPE uses frequency and merge-rule replay; WordPiece uses likelihood and MaxMatch lookup.
• Next: SentencePiece (raw-text subwords, no pre-tokenization) → Tokenization Pipeline → Token IDs.