Token Embeddings

A token embedding is the learned vector that a model assigns to one entry of its vocabulary. The model reads it from the embedding matrix by using the token ID as a row index.

A token ID is just an integer used to identify a token in the model's vocabulary. For example (illustrative IDs):

• cat → token ID 3797
• dog → token ID 8531

The numbers themselves don't inherently express meaning. The ID for dog being larger than the ID for cat does not mean a dog is more important, bigger, or more similar to a cat. The numbering follows the order in which the vocabulary was built — nothing more.

The Token IDs page used a library analogy: an ID is a call number. This page continues it.

• Tokenization hands the model a slip of paper with call numbers: [464, 3797, 318, 10447].
• Embedding lookup is the trip to the shelf. The call number is only used to find the right spot; what comes back is the book — a long learned vector.
• The transformer is the scholar who reads those books together and works out how each one is being used in this particular sentence.

Two tables, one ID

A token ID works like a row ID. The same number is used as the key into two different tables. (Small IDs below are illustrative.)

• The vocabulary is a dictionary-like mapping ("cat" → 1), usually a hash map, and often supports ID → token as well.
• The embedding matrix is not a dictionary. It is a dense array indexed by position, E[1]. It stores no token strings.
• They are aligned only by the shared number. Row i of E belongs to token i because training used that alignment. Change the tokenizer and every ID points at the wrong row.
• Row count vs vocabulary size: usually one row per vocabulary entry, though some models pad E with extra rows.

Does a token ID contain the vector numbers?

No. The ID only says which row to read. Take token ID 1 (cat) from the tables above, with a 4-number row (illustrative values):

The ID 1 is not the vector [0.80, −0.30, 0.40, 0.15]. It is the address used to retrieve it.

Where do the vector numbers come from?

• Start: the embedding values are typically initialized with a numerical initialization scheme, so at first they carry no useful meaning.
• Training: the model adjusts them through optimization and backpropagation (see phase 04).
• Result: over time the values become useful for the model's language-processing task. A real LLM stores hundreds or thousands of numbers per token.

Remember: token ID = which row?  Vector = the learned numbers stored in that row.

It is tempting to say "the model learns the meaning at the embedding step". That is not quite right. The embedding step only reads parameters that were already learned during training.

Where the knowledge actually lives

• The embedding matrix and the transformer weights were established during training.
• At inference the model uses those fixed parameters to process your input. It does not suddenly acquire knowledge at the embedding stage.
• The embedding matrix typically starts with random values and is adjusted by backpropagation until related tokens end up with useful vectors.

Follow "The cat is sleeping" through the pipeline:

graph LR
    A["Text: The cat is sleeping"] --> B["Token IDs: 464, 3797, 318, 10447"]
    B -->|"row lookup in E"| C["4 vectors, each D numbers"]
    C --> D["Transformer layers"]
    D --> E["Contextual representations"]

    style A fill:#3b82f6,stroke:#1d4ed8,color:#fff
    style B fill:#10b981,stroke:#047857,color:#fff
    style C fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style E fill:#f59e0b,stroke:#b45309,color:#fff
    

Real models use hundreds or thousands of dimensions. Here we use 4 so every number fits on the page. The vectors are invented for teaching; they are not from a trained model.

What does "dimension" mean?

Dimension is how many numbers are inside one token's vector. Using cat from Step 1 (illustrative values):

Each dimension is one position in the vector. Together the numbers form the token's learned numerical representation. Language has many patterns and relationships, and a single number would give the model very little capacity to represent them.

A human analogy

• Person → the thing being represented (here: the token).
• Measurements → the individual dimensions.
• List of measurements → the vector.
• Number of measurements → the dimensionality.

How big are vectors in real models?

These are example sizes; actual models vary. With 4,096-dimensional embeddings, each token ID retrieves a vector of 4,096 numbers. For a matrix with 50,000 tokens and 4,096 dimensions, the shape is 50,000 × 4,096: 50,000 rows, each containing 4,096 numbers.

A more detailed person description can capture more detail, and likewise more dimensions can represent more. But more dimensions alone do not guarantee a better model; architecture, training and the learned parameters matter too.

Let $V$ be the vocabulary size and $D$ the embedding dimension. The embedding matrix is

Same thing as a one-hot matrix product

If $\mathbf{o}_i \in \{0,1\}^{V}$ is the one-hot vector with a 1 at position $i$, then

So the lookup is mathematically a matrix product, but frameworks implement it as direct row indexing because multiplying by a mostly-zero vector would waste work.

Size

Pick any two words. The lab compares what the IDs say with what the vectors say, using the toy 4-dimensional matrix from the Calculate phase.

Part A — pure Python:

import math

vocab = {"The": 464, "cat": 3797, "is": 318, "sleeping": 10447, "dog": 8531}  # illustrative IDs

E = {   # toy 4-d "embedding matrix", keyed by token ID
    464:   [ 0.10, -0.05,  0.02,  0.30],
    3797:  [ 0.21, -0.43,  0.87,  0.15],
    318:   [ 0.05,  0.02, -0.04,  0.22],
    10447: [-0.30,  0.55,  0.40, -0.10],
    8531:  [ 0.25, -0.40,  0.80,  0.10],
}

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

ids = [vocab[w] for w in ["The", "cat", "is", "sleeping"]]
print(ids)                       # [464, 3797, 318, 10447]

vectors = [E[i] for i in ids]    # the embedding lookup
print(vectors[1])                # [0.21, -0.43, 0.87, 0.15]

print(abs(vocab["cat"] - vocab["dog"]))            # 4734  (ID gap)
print(abs(vocab["cat"] - vocab["is"]))             # 3479  (ID gap)
print(round(cosine(E[3797], E[8531]), 3))          # 0.997 (cat vs dog)
print(round(cosine(E[3797], E[318]), 3))           # 0.0   (cat vs is)

Part B — PyTorch nn.Embedding:

import torch
import torch.nn as nn

emb = nn.Embedding(num_embeddings=50257, embedding_dim=768)   # E has shape [50257, 768]

ids = torch.tensor([[464, 3797, 318, 10447]])   # shape [batch=1, seq_len=4]
x = emb(ids)                                     # row lookup: weight[ids]
print(x.shape)                                   # torch.Size([1, 4, 768])

print(emb.weight.requires_grad)                  # True  -> the rows are learned parameters
print(emb.weight.numel())                        # 38597376

Try these with a real tokenizer and model (for example gpt2 from HuggingFace):

• First layer of every language model: an integer tensor [B, L] becomes a float tensor [B, L, D].
• Position is added separately. The lookup says nothing about word order; models add positional information (for example GPT-2 adds learned position embeddings) before the first transformer layer.
• Weight tying: many models, including GPT-2, reuse the input embedding matrix as the output projection that scores the next token.
• Vocabulary changes: when you add tokens during fine-tuning, you must resize the embedding matrix (for example model.resize_token_embeddings(len(tokenizer))) so each new ID has a row.
• Memory: for small models the embedding matrix can be a large share of all parameters.

🚫 Common Misconceptions to Correct

• Misconception: a bigger ID means a more important word.
  Reality: IDs are labels. Importance or similarity is expressed, if at all, by the vectors.
• Misconception: the embedding lookup does computation.
  Reality: It is a constant-time row read. The learning happened earlier, during training.
• Misconception: after tokenization the model already understands the sentence.
  Reality: After tokenization it has identifiers; after embedding it has vectors; only after the transformer layers does it have context-aware representations.

📌 Summary — Everything in Six Bullets

• IDs identify, vectors represent. Tokenization yields integers; the transformer needs numeric vectors.
• Lookup, not computation. Token ID i selects row i of the embedding matrix E of shape [V, D].
• IDs carry no usable geometry. The size of an ID or the gap between two IDs means nothing.
• Knowledge lives in learned weights. The embedding matrix and transformer weights come from training; the lookup merely reads them.
• Token embeddings are context-free. The same token gives the same input vector in every sentence.
• Pipeline: Text → Token IDs → Embedding vectors → Transformer computations → Contextual representations.