Interactive Visual Lab #charan

Embedding Lookup & Space Simulator

Follow a Token ID into the Embedding Matrix (W_E), watch its row get copied out as a learned vector, stack the vectors into a sequence matrix, see where tokens land in embedding space, measure closeness with cosine similarity, and finally see why a static embedding still needs attention to become contextual.

Presets:

Speed

Amber cell = positive value Cyan cell = negative value 💡 Click any token chip or dot to inspect its embedding

Selected Token

Charan

Token ID (Row Index)

7490

Embedding (first 3 dims)

[…]

Nearest Token (cosine)

—

🧮 What is the Embedding Matrix really? #charan

A giant table of learned weights: W_E has shape [vocab_size × d_model], e.g. 50,257 × 768 for GPT-2 — about 38.6 million numbers.

Lookup, not computation: e = W_E[token_id]. The Token ID is only a row index. Nothing is calculated at this stage.

No human-readable dimensions: dimension 3 is not “height” and dimension 5 is not “colour”. Training nudges every number until the whole model predicts language well.

⚡ The Big Hand-off: Embeddings $\to$ Position $\to$ Attention #charan

Static by design: “plant” gets the exact same vector in “I plant an idea” and “the green plant needs water”.

Positional information is added next so the model also knows word order.

Self-Attention then mixes each vector with its neighbours, producing a context-aware vector — that is where real meaning is resolved.

← Back to Embeddings Overview 📝 Charan's Embeddings Notes ⚡ Tokenization Simulator 🚀 15-Stage Transformer Simulator →