SentencePiece

SentencePiece is a tokenizer framework, not a single algorithm. Created by Google (Kudo & Richardson, 2018), it can run BPE or Unigram Language Model as its internal subword algorithm. The key innovation: it treats raw text as a stream of Unicode characters, requiring no language-specific pre-tokenization.

Standard BPE and WordPiece rely on pre-tokenization — splitting text on whitespace before learning subwords. This creates fundamental problems:

• Language coverage: Chinese, Japanese, Thai, and many other languages have no whitespace between words. A whitespace split produces single characters or meaningless units.
• Reproducibility: different platforms may handle whitespace and Unicode normalization differently. SentencePiece guarantees byte-identical output everywhere given the same .model file.
• Pipeline complexity: every pre-tokenization choice (split on spaces? on punctuation? strip accents?) is a potential source of inconsistency. SentencePiece removes all those choices.

Analogy: Standard BPE is like a recipe that says "first, chop the onions" — it assumes you have pre-prepped, whitespace-separated words before it starts. SentencePiece starts from the whole vegetable — raw text — and does all the chopping itself.

The ▁ character (a small underline, U+2581) is the key. SentencePiece replaces every whitespace with ▁ and attaches it to the beginning of the following piece. This way:

• The raw text can be reconstructed exactly from pieces by replacing ▁ with a space.
• Word boundaries are encoded within the piece stream, not by the structure of the input.
• For languages without spaces (Chinese, Thai…), ▁ simply doesn't appear — the algorithm still works identically.

Standard BPE workflow:
  raw text → [SPLIT on whitespace] → words → BPE merge → pieces

SentencePiece workflow:
  raw text → [replace spaces with ▁] → character stream → BPE/Unigram → pieces

"Hello world" (SP):
  spaces become ▁ → "▁Hello▁world" → ["▁Hello", "▁world"]

"世界" (SP, no spaces):
  no ▁ injected → "世界" → ["▁世界"] or ["▁世", "界"] depending on vocab

A. SentencePiece + BPE mode

• Same merge-based vocabulary learning as standard BPE, but applied to the raw Unicode character stream (with ▁ inserted for spaces).
• Deterministic: given the same corpus and settings, training always produces the same vocabulary and the same tokenization.
• Used by: LLaMA (v1/v2), mBART.

B. SentencePiece + Unigram Language Model mode

• Instead of greedily merging pairs, Unigram starts with a large vocabulary and prunes it by removing pieces whose removal decreases corpus log-likelihood the least (top-down).
• Assigns a probability to each tokenization; picks the most likely one at inference via Viterbi search.
• Can sample multiple tokenizations during training as a regularization technique (subword regularization).
• Used by: T5, mT5, ALBERT.

SentencePiece manages the full pipeline internally. You provide a raw text corpus and a config; it handles everything from space encoding to vocabulary output.

Training

# Training config (SentencePiece CLI or Python API)
model_type         = "bpe"     # or "unigram"
vocab_size         = 32000     # target vocabulary size
character_coverage = 0.9995   # fraction of unique chars to cover
                               # lower → more byte-fallback for rare chars
input              = "corpus.txt"   # raw text, any language
model_prefix       = "mymodel"     # produces mymodel.model + mymodel.vocab

# Output files:
#   mymodel.model  — binary protobuf (portable, platform-independent)
#   mymodel.vocab  — human-readable vocabulary with log-probabilities

Inference (single-pass)

Concrete piece outputs across three example inputs, with ▁ markers highlighted:

Example 1 — basic English
  Input:  "hello world"
  Pieces: ["▁hello", "▁world"]   ← ▁ on every word-start piece
  IDs:    [12345,    6789]        ← example IDs only

Example 2 — mixed script (no failure, no [UNK])
  Input:  "hello世界"
  Pieces: ["▁hello", "世界"]

Example 3 — punctuation and contractions
  Input:  "It's a test."
  Pieces: ["▁It", "'", "s", "▁a", "▁test", "."]
  # ▁ only on pieces that start a new word

Example 4 — reconstructing original text
  Pieces: ["▁It", "'", "s", "▁a", "▁test", "."]
  → join:          "▁It's▁a▁test."
  → replace ▁→' ': " It's a test."
  → strip leading: "It's a test."  ✓

Standard BPE has an extra pre-tokenization step that SentencePiece eliminates entirely:

graph LR
    A["Raw text: 'Hello world'"] -->|replace spaces with ▁| B["▁Hello▁world"]
    B -->|BPE or Unigram| C["Pieces: ['▁Hello','▁world']"]
    C -->|join + ▁→space| D["Decoded: 'Hello world' ✓"]

    style A fill:#3b82f6,stroke:#1d4ed8,color:#fff
    style B fill:#f59e0b,stroke:#b45309,color:#fff
    style C fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style D fill:#34d399,stroke:#047857,color:#fff
        

Try the live SentencePiece-style simulation. Type any text and see how spaces become ▁ markers on the first piece of each word:

T5 (and mT5, ALBERT) use SentencePiece under the hood. Load the tokenizer and observe the ▁ marker directly:

from transformers import T5Tokenizer

tok = T5Tokenizer.from_pretrained("t5-small")

# Basic English
print(tok.tokenize("Hello world"))
# ['▁Hello', '▁world']

# Mixed script — no failure, no [UNK]
print(tok.tokenize("hello世界"))
# ['▁hello', '世', '界']

# Punctuation and contractions
print(tok.tokenize("It's a test."))
# ['▁It', "'", 's', '▁a', '▁test', '.']
#   ↑ only word-start pieces carry ▁

# Reconstruction — always use the tokenizer, never "".join()
pieces = tok.tokenize("Hello world")
print(tok.convert_tokens_to_string(pieces))
# 'Hello world'  ← ▁ → space, leading space stripped

• T5 and mT5: Google's text-to-text transfer transformer family. mT5 is trained on 101 languages — SentencePiece's language-agnostic tokenization is essential here.
• ALBERT: Uses SentencePiece + Unigram with 30,000 vocabulary. The Unigram mode's probabilistic sampling is used for subword regularization during training.
• LLaMA (v1 and v2): Meta's LLaMA uses SentencePiece + BPE with 32,000 vocabulary. The ▁ markers are visible if you inspect its tokenizer output.
• mBART: Multilingual sequence-to-sequence model — SentencePiece ensures consistent tokenization across all 25 languages without 25 language-specific pre-tokenization rules.

• SentencePiece is NOT an algorithm. Saying "model X uses SentencePiece" doesn't tell you what subword algorithm it uses. You need to check BPE vs Unigram mode explicitly.
• ▁ means non-trivial reconstruction. Pieces cannot be concatenated with "".join(). Code that assumes simple joining will silently produce garbage output with ▁ characters embedded.
• Byte-fallback surprises. When character_coverage < 1.0, rare characters are encoded as UTF-8 byte pieces (e.g. <0xE4>). These look like HTML entities but are byte fallbacks — they will confuse users who haven't encountered them.
• Unigram mode is non-deterministic at training time. The pruning algorithm involves randomness. Two training runs on the same corpus may produce slightly different vocabularies. Fix the random seed for reproducibility.
• Not interchangeable. A SentencePiece model trained for T5 cannot be used with BERT. Vocabularies, IDs, and normalization schemes are all different.

📌 Summary — five things to remember

• Framework, not algorithm: SentencePiece wraps BPE or Unigram — it is not a new tokenization algorithm in itself.
• No pre-tokenization: treats raw Unicode text as a stream; no language-specific whitespace splitting required.
• ▁ replaces whitespace: spaces become the ▁ (U+2581) prefix on word-start pieces, encoding word boundaries without splitting the input first.
• BPE or Unigram inside: BPE mode is merge-based and deterministic; Unigram mode is probabilistic and supports subword regularization during training.
• Used by multilingual models: T5, mT5, ALBERT, LLaMA (v1/v2), mBART — whenever cross-language consistency matters, SentencePiece is the standard choice.