🌍 SentencePiece - Language-Agnostic Tokenization

Used by: T5, XLNet, ALBERT, mT5, LLaMA, and most multilingual models

🎯 Why SentencePiece?

The Problem with Traditional Tokenizers

English: "Hello world" → ["Hello", "world"] ✅ (spaces separate words)
Chinese: "你好世界" → ["你好世界"] ❌ (no spaces!)
Japanese: "こんにちは世界" → ["こんにちは世界"] ❌ (no spaces!)

Traditional tokenizers need language-specific rules.
SentencePiece: Treats everything as raw bytes → Works for ALL languages!

SentencePiece Advantages

1. Language-Agnostic - No preprocessing needed
2. Reversible - Perfect text reconstruction
3. Flexible - Supports BPE, Unigram algorithms
4. Production-Ready - Fast C++ implementation
5. Multilingual - Single model for all languages

🔬 How SentencePiece Works

Core Principles

1. Treat input as raw byte sequence
2. Add ▁ character to mark word boundaries
3. Train subword model (BPE or Unigram)
4. Tokenize without preprocessing

Visual Example

Traditional:
"Hello world" → Pre-tokenize → ["Hello", "world"] → Tokenize

SentencePiece:
"Hello world" → "▁Hello▁world" → ["▁Hello", "▁world"]
(▁ = space marker)

💻 SentencePiece in Practice

Training a Model

import sentencepiece as spm

# Train model
spm.SentencePieceTrainer.train(
    input='corpus.txt',
    model_prefix='m',
    vocab_size=1000,
    model_type='bpe',  # or 'unigram'
    character_coverage=0.9995,
)

Using the Model

import sentencepiece as spm

sp = spm.SentencePieceProcessor()
sp.load('m.model')

text = "Hello world! 你好世界"
tokens = sp.encode_as_pieces(text)
ids = sp.encode_as_ids(text)

print(f"Tokens: {tokens}")
print(f"IDs: {ids}")

🎯 SentencePiece vs Others