One-Hot Encoding

One-hot encoding is the most direct answer to the question: how do you give a model an unambiguous identity for each category in a set? Given a vocabulary of size V, each token or category is represented as a vector of length V where every position is 0 except for exactly one position — the one corresponding to this token — which is 1.

The name comes from digital electronics: a "one-hot" signal is a bus where exactly one wire is high (1) at any given time. In NLP, the "bus" is the vocabulary, and the "high wire" is the token's unique index.

Vocabulary: [cat, dog, fish, bird]   (V = 4)

cat  → [1, 0, 0, 0]
dog  → [0, 1, 0, 0]
fish → [0, 0, 1, 0]
bird → [0, 0, 0, 1]

The naive alternative — assigning integer IDs (cat=0, dog=1, fish=2, bird=3) and feeding those integers directly into a model — creates an implicit ordering that does not exist. If dog gets ID 1 and fish gets ID 2, the model may infer that fish is "greater than" or "closer to 3" than dog, which is nonsensical. One-hot eliminates this fake numeric order entirely.

Imagine a control panel with V switches, one for each word in your vocabulary. Exactly one switch is ON at any time — the switch labelled with the current token. Every other switch is OFF. The entire state of the panel is your one-hot vector.

The key insight: the position of the 1 carries all the information. The value of the 1 is always just 1 — it signals presence, not magnitude. There is no "more cat-ness" or "less dog-ness" in this encoding.

This is fundamentally different from, say, word frequency counts, where values vary in magnitude and carry information. In one-hot, every active token has exactly the same activation value: 1.

One-hot vectors form the standard basis of an V-dimensional space. Each vector $\mathbf{e}_i$ has a 1 at index $i$ and 0s everywhere else. This has precise geometric consequences.

The geometric consequence of equidistance is that one-hot vectors encode no semantic topology. "cat" and "dog" are no closer together than "cat" and "spaceship". The encoding is maximally agnostic about semantic relationships — it delegates all relationship learning to the embedding layer on top.

Constructing a one-hot vector for a token is entirely mechanical — no learning, no parameters, no ambiguity:

1. Fix the vocabulary: Enumerate all possible tokens and assign each a unique integer ID. The order is arbitrary but must remain fixed. Vocabulary size is V.
2. Find the token's ID: Look up the token in the vocabulary dictionary to retrieve its integer index i.
3. Create a zero vector: Allocate a vector of length V filled with zeros: [0, 0, …, 0].
4. Set position i to 1: Flip exactly one position to 1: vector[i] = 1. The result is the one-hot encoding.

The embedding layer in a neural network performs the equivalent of: take the one-hot vector $\mathbf{e}_i$, multiply by the embedding matrix $W \in \mathbb{R}^{V \times d}$, and get the $d$-dimensional embedding: $W^\top \mathbf{e}_i = W[i, :]$ — the $i$-th row of $W$. In practice, this is implemented as a direct row index rather than a full matrix multiply, but the math is identical.

Walk through a complete manual construction. Vocabulary: [red, green, blue, yellow, black] — V = 5. Token to encode: "blue".

Vocab [red, green, blue, yellow, black]  →  IDs [0, 1, 2, 3, 4]

red    → [1, 0, 0, 0, 0]
green  → [0, 1, 0, 0, 0]
blue   → [0, 0, 1, 0, 0]   ← token "blue" → ID 2 → position 2 is hot
yellow → [0, 0, 0, 1, 0]
black  → [0, 0, 0, 0, 1]

Dot product check: blue · green = [0,0,1,0,0] · [0,1,0,0,0] = 0+0+0+0+0 = 0. Orthogonal, as expected.

Distance check: ‖blue − green‖₂ = ‖[0,−1,1,0,0]‖₂ = √(0+1+1+0+0) = √2 ≈ 1.414. Equidistant, as expected.

Select any token from the demo vocabulary below and click Encode to see its one-hot vector rendered as a grid of cells. The blue cell marks the active dimension — every other position is zero.

Now try to observe these properties as you switch tokens:

• The position of the hot cell changes but its value is always 1.
• All other cells are always 0 — the vector is maximally sparse.
• Two different tokens can never share the same hot position.
• Adding a new token to the vocabulary would grow every vector by one dimension.

A fully annotated implementation showing one-hot construction from scratch, with numpy for efficiency, and a demonstration of the key properties (orthogonality, equidistance, and the embedding-lookup equivalence).

import numpy as np

# ── Step 1: Build vocabulary ────────────────────────────────────────────────
vocab = ["cat", "dog", "fish", "bird", "rabbit"]
token_to_id = {tok: i for i, tok in enumerate(vocab)}
V = len(vocab)

# ── Step 2: Core one-hot function ───────────────────────────────────────────
def one_hot(token_id: int, vocab_size: int) -> np.ndarray:
    """
    Create a one-hot vector of length vocab_size.

    Args:
        token_id:   Integer index of the token (0-based).
        vocab_size: Total vocabulary size V.

    Returns:
        numpy array of shape (V,) with a single 1.0 at position token_id.
    """
    vec = np.zeros(vocab_size, dtype=np.float32)
    vec[token_id] = 1.0
    return vec

# ── Step 3: Encode tokens ───────────────────────────────────────────────────
cat_vec  = one_hot(token_to_id["cat"],  V)   # [1, 0, 0, 0, 0]
dog_vec  = one_hot(token_to_id["dog"],  V)   # [0, 1, 0, 0, 0]
fish_vec = one_hot(token_to_id["fish"], V)   # [0, 0, 1, 0, 0]

# ── Step 4: Verify geometric properties ────────────────────────────────────
print(f"cat · dog  = {np.dot(cat_vec, dog_vec):.1f}")          # 0.0 (orthogonal)
print(f"cat · cat  = {np.dot(cat_vec, cat_vec):.1f}")          # 1.0 (self-dot)
print(f"‖cat−dog‖  = {np.linalg.norm(cat_vec - dog_vec):.4f}") # 1.4142 (√2)
print(f"‖cat−fish‖ = {np.linalg.norm(cat_vec-fish_vec):.4f}")  # 1.4142 (√2, always)
print(f"Non-zero entries: {int(cat_vec.sum())}")                 # 1

# ── Step 5: The embedding lookup equivalence ────────────────────────────────
d = 4   # small embedding dimension
np.random.seed(0)
embedding_matrix = np.random.randn(V, d).astype(np.float32)

# Method A: matrix multiply with one-hot (expensive — O(V*d))
embed_A = embedding_matrix.T @ cat_vec           # shape: (d,)

# Method B: direct index lookup (efficient — O(d))
embed_B = embedding_matrix[token_to_id["cat"]]   # shape: (d,)

print(f"Methods match: {np.allclose(embed_A, embed_B)}")  # True

Run these experiments in a Python notebook or your browser console. Each one challenges a common misconception.

These three representations sit at successive levels of abstraction in the token → model pipeline. Understanding their relationship is foundational:

• Memory does not scale: For V = 100,000 tokens, each one-hot vector has 100,000 elements with 99,999 of them being zero. Storing a corpus of 1 billion tokens as dense one-hot would require ~400 GB of float32 data.
• Every token is equidistant: Cat and dog, cat and democracy, cat and the number 7 — all are distance √2 from each other. One-hot encodes no topology of meaning, no clusters, no analogies.
• No learned relationships: Synonyms, antonyms, hypernyms, analogies — none of these are captured. The encoding is purely identity-based; relationships must be learned on top (via embeddings).
• Vocabulary must be fixed in advance: Adding a new token requires extending every previously encoded vector's dimension. This makes one-hot brittle for open-vocabulary or streaming settings.

Misconceptions to address:

• "One-hot vectors encode similarity" — No: all distinct tokens are equidistant (dot product = 0, distance = √2).
• "A larger ID means a more important word" — No: IDs are arbitrary indices with no semantic ordering.
• "One-hot is what language models actually use internally" — No: LLMs use dense learned embeddings; one-hot is the conceptual foundation, not the runtime representation.
• "I need to materialize the full one-hot vector in my code" — No: embedding_matrix[token_id] is mathematically equivalent and uses 99%+ less memory.

• What it is: A V-dimensional binary vector with exactly one 1, at the token's vocabulary index.
• Why it exists: Prevents fake numeric ordering of raw integer IDs; gives each category its own orthogonal axis.
• Geometry: All distinct tokens are perfectly orthogonal (dot = 0) and equidistant (distance = √2).
• Memory: Too large to materialize at scale — V=50k means 200 KB per vector; replaced by direct index lookup.
• Relationship to embeddings: An embedding lookup is mathematically W.T @ one_hot — the embedding matrix projects from one-hot space to a learned dense space.
• Limitation: Encodes zero semantic similarity — motivates the move to learned dense embeddings.