Subword boundaries are learned from training text. Tokenizers can split unfamiliar words, but token counts and coverage vary by vocabulary and language.
③ Token IDs
Each token maps to a model-specific integer ID
Token → ID mapping
GPT-2-style vocabulary lookup · 50,257 tokens
Charan only sees:
Vocabulary board (sample)
Matching entries lit in purple
④ Embeddings
Token IDs → dense 768-dimensional input embeddings (GPT-2 small example)
Vector for "lost"
768 values — illustrative input embedding
Embedding matrix — all tokens
Each row = one token's 768-dim input embedding
"lost cities" and "forgotten kingdoms" cluster nearby — both relate to historical disappearance. "archaeological" sits close to both.
⑤ Self-Attention
Tokens talk to each other — focus assigned
Illustrative attention heatmap
Brighter = stronger relationship
What "mysteries" attends to
Illustrative context weights
This heatmap and these weights are illustrative, not model activations. A real attention pattern must be extracted from a specified model and input.
⑥ Transformer Layers
GPT-2 small example · 12 transformer blocks
Layer processing
Layer 0 / 12
What each depth learns
Emergent understanding
Layers 1–4 — Different patterns can emerge; not a fixed rule Layers 5–8 — Representations combine information; behavior varies by model Layers 9–11 — Later blocks further transform representations Final block — Output representation is used to compute next-token scores
⑦ Generation
One token at a time — sampled from probability distribution