Embeddings

An embedding is the dense vector a token retrieves from a learned lookup table. When a model processes the word cat, it first converts it to a token ID (say, 4127), then looks up row 4127 in a large matrix called the embedding table. The result is a compact list of floating-point numbers — a vector — that the model uses as the token's representation.

Raw token IDs are just arbitrary integers. Token ID 4127 has no mathematical relationship to token ID 4128 — the numbers are random addresses, not magnitudes. Feeding raw IDs to a neural network layer would impose a false ordering on tokens that have none.

• IDs have no magnitude meaning: ID 9000 is not "more" of anything than ID 1. Multiplying by an ID number is meaningless.
• One-hot vectors are vocab-sized: A 50,000-token vocabulary means 50,000-dimensional vectors. Every dot product involves 50,000 multiplications — almost all of which are multiplying by 0. Grossly wasteful.
• Dense embeddings are compact: 768 floats per token. 98.5% less data. 100% non-zero. Every dimension contributes to every computation.
• Dense embeddings are learnable: The embedding table is a parameter matrix — it starts random and updates during training to encode useful patterns.

The embedding table is a 2D matrix with one row per vocabulary token and one column per embedding dimension. The shape is $E \in \mathbb{R}^{V \times d}$, where:

• $V$ = vocabulary size (e.g. 50,257 for GPT-2; 30,522 for BERT-base)
• $d$ = embedding dimension (e.g. 768 for BERT-base, 1024 for GPT-2 medium)
• $x_i = E[t_i]$ — the embedding for token $t_i$ is the row at index $t_i$

Here is what the table looks like conceptually. Each row is the full $d$-dimensional vector for one token:

       dim₀    dim₁    dim₂    dim₃   …   dim₇₆₇
row 0  [PAD]  0.00    0.00    0.00    0.00   …   0.00
row 101 [CLS]  0.34   -0.12    0.87   -0.45   …   0.21
row 102 [SEP] -0.11    0.56   -0.23    0.78   …  -0.09
row 1996 cat    0.12   -0.41    0.77    0.03   …   0.55
row 3899 dog    0.14   -0.38    0.74    0.07   …   0.52
row 2482 car   -0.65    0.22   -0.31    0.88   …  -0.44
   …     …       …       …       …     …    …

Embedding lookup is deliberately simple: it is a table access, not a computation. No arithmetic happens at lookup time — the model just reads the pre-stored row.

Every embedding is a point in $\mathbb{R}^d$. Once tokens are embedded, all the geometric tools you've studied in this roadmap branch apply directly and are used constantly inside the model:

Embeddings are commonly misunderstood. Here is a precise list of what they are and what they are not:

Start with pure Python to see that embedding lookup is nothing more than array indexing, then see the PyTorch equivalent that adds gradient tracking and batch support:

import random

# ── Pure Python: embedding table as a 2D list ────────────────────────────────
VOCAB = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "cat": 4, "dog": 5, "car": 6}
V = len(VOCAB)   # vocabulary size
D = 8            # embedding dimension (tiny, for illustration)

random.seed(42)
# Each row is a token's embedding — initialised uniformly at random
E = [[round(random.uniform(-1, 1), 3) for _ in range(D)] for _ in range(V)]

def embed(token: str) -> list[float]:
    """Look up the embedding for a single token."""
    idx = VOCAB.get(token, VOCAB["[UNK]"])   # unknown tokens → [UNK] row
    return E[idx]                             # row selection — O(1) table lookup

print(embed("cat"))   # e.g. [0.304, -0.719, 0.218, ...]
print(embed("dog"))   # different values, but after training would be similar to cat

# ── PyTorch: nn.Embedding (what production models use) ───────────────────────
import torch
import torch.nn as nn

V_real = 50257  # GPT-2 vocabulary size
D_real = 768    # GPT-2 small embedding dimension

# Create the embedding table — weights initialised randomly, updated by backprop
embedding_table = nn.Embedding(num_embeddings=V_real, embedding_dim=D_real)

# Simulate a token ID sequence: ["cat", "sat", "on", "the", "mat"]
token_ids = torch.tensor([4127, 2938, 2006, 1996, 13523])   # hypothetical IDs

# Embedding lookup: each ID → corresponding row → shape (5, 768)
vectors = embedding_table(token_ids)
print(vectors.shape)   # torch.Size([5, 768])

# nn.Embedding is differentiable — gradients flow back into the table during training
# The table's .weight attribute is the V×D matrix, accessible for inspection:
print(embedding_table.weight.shape)   # torch.Size([50257, 768])

This page gives you the vector-world view of embeddings: what the table is, what the shape means, how lookup works, and how it connects to distance, similarity, and attention. That is enough to understand how vectors flow through a transformer.

Also worth knowing what comes immediately after embeddings in the transformer pipeline:

Common misconceptions to tackle:

• "Embeddings are the word's meaning" — No: they are learned numeric patterns that help a specific model on a specific task. "Meaning" is a human concept; embeddings approximate certain aspects of it.
• "I can swap embedding tables between models" — No: different models have different vocabularies, different dimensions, and different geometric structures. They are not interchangeable.
• "Bigger embedding dimension = better meaning" — Not necessarily. d=64 may be enough for a small task; d=4096 is needed for GPT-4. The right size depends on model scale and task complexity.
• "Token IDs and embeddings are the same thing" — No: a token ID is an integer address. An embedding is the dense float vector stored at that address. One is a key; the other is the value.

Summary