Tokenization & BPE Splitter Simulator
Watch how raw human language gets sliced, normalized with whitespace markers (␣), merged via Byte Pair Encoding (BPE), and assigned immutable integer Token IDs ready for Embedding Matrix lookup.
Selected Token
Charan
Token ID (Integer)
7490
Type
Whole Word
Next Destination
Embedding Row #7490
🧩 Why Subwords instead of Characters or Words? #charan
1. Characters (26 letters): Sequences are too long, attention matrices explode in memory ($O(N^2)$), and characters hold zero inherent semantic meaning.
2. Full Words: Vocabulary would need to be infinite to cover every plural, conjugation, typo, and rare name (e.g. “Advaitha”).
3. BPE Subwords (Sweet Spot): Frequent words stay intact (“planted”), while rare words decompose gracefully into familiar subwords (“Adva” + “itha”), completely eliminating Out-Of-Vocabulary (OOV) errors.
⚡ The Big Hand-off: Tokens $\to$ Embeddings $\to$ Attention #charan
Tokenizer's Sole Job: Turn human text into an array of integers [7490, 12785, 257, ...]. The tokenizer has no understanding of meaning or context.
Embedding Matrix ($W_E$): Uses each Token ID as a row selector to pull out its static feature vector.
Multi-Head Attention: Mixes and transforms those static numbers into dynamic Context Vectors where “plant” adapts its numbers based on surrounding tokens!