Token IDs

A token ID is a single integer. It is the row index of a token in the model's embedding matrix. The tokenizer converts text to a list of token strings, then maps each string to its integer ID. The model never sees the string — only the number.

Neural networks operate on floating-point tensors. There is no operation "multiply a string by a matrix." You need a way to turn a discrete category (the token) into a number so the model can retrieve its learned representation.

• A matrix multiply requires numeric input — strings are meaningless to PyTorch.
• IDs are the categorical lookup keys that map each token to a row in the embedding matrix.
• Without IDs, there is no addressable way to retrieve the learned float vector for a token.
• IDs let you represent a full sentence as a compact integer array — far more efficient than storing strings in memory.

Token IDs are like library call numbers in the Dewey Decimal System. The number 791.43 tells you the shelf where films are shelved. It does not mean films are 791.43 times anything. It is just an address.

Token ID 500 is not "more meaningful" or "more important" than ID 20. It is not five times anything. It just lives on a different shelf — row 500 versus row 20 in the embedding matrix. The number tells you where to look, never what to think about what you find.

The vocabulary is a bijective mapping (one-to-one, onto) between token strings and integers:

• IDs start from 0. They are contiguous integers: 0, 1, 2, …, vocab_size−1.
• Special tokens ([PAD], [UNK], [BOS], [EOS], [MASK], etc.) occupy reserved IDs, usually at the low end.
• The embedding matrix has shape [vocab_size × embed_dim]. Row N is the learned float vector for token ID N.
• Accessing row N: embedding_matrix[N] returns a vector of embed_dim floats (e.g. 768 for BERT-base).

There are exactly two operations on the vocabulary mapping:

• encode (token_string → ID): look up the string in the vocab dict. If missing, return the ID of [UNK].
• decode (ID → token_string): look up the integer in the id_to_token reverse dict.

Build a toy vocabulary with 5 entries, then encode and decode by hand. Follow each step carefully.

The embedding matrix $\mathbf{E}$ has shape $[V, D]$ where $V$ is the vocabulary size and $D$ is the embedding dimension. A token ID is simply a row selector.

Vocabulary row vs embedding row

A token ID can be treated as a row ID in two tables. The numbers below are illustrative.

• The vocabulary maps tokens to IDs (and often back). The embedding matrix maps IDs to learned vectors.
• The matrix stores no token strings; it lines up with the vocabulary only through the shared ID.
• The ID is the key. The learned vector is the content, and it is what carries the numerical representation. See Token Embeddings for the full walkthrough.

Does a token ID contain the vector numbers?

No. A token ID identifies which row of the embedding matrix to read. The numbers stored in that row form the token's vector. Using the toy 4-number row for cat from the table above (illustrative values):

Sample vocabulary entries showing the ID ↔ token mapping:

🔄 Encode / Decode Flow

graph LR
    A["String: 'AI'"] -->|Vocabulary Map| B["Token ID: 9821"]
    B -->|Embedding Table Index| C["Row #9821 of E"]
    C -->|Output| D["Vector: [0.15, -0.92, ..., 0.38]"]
    D -->|argmax during generation| B2["Token ID: 9821"]
    B2 -->|Reverse vocab lookup| A2["String: 'AI'"]

    style A fill:#3b82f6,stroke:#1d4ed8,color:#fff
    style B fill:#10b981,stroke:#047857,color:#fff
    style D fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style B2 fill:#10b981,stroke:#047857,color:#fff
    style A2 fill:#3b82f6,stroke:#1d4ed8,color:#fff
          

Part A — toy vocabulary in pure Python:

# ── Toy vocabulary ────────────────────────────────────────────────
vocab = {
    "[PAD]": 0, "[UNK]": 1, "[BOS]": 2, "[EOS]": 3,
    "I": 4, "love": 5, "AI": 6, "and": 7, "learning": 8,
    "the": 9, "of": 10, "to": 11, "a": 12, "is": 13,
    "tokens": 14, "model": 15, "text": 16, "word": 17,
}

id_to_token = {v: k for k, v in vocab.items()}   # reverse map

def encode(sentence: str) -> list[int]:
    """Whitespace-tokenize and look up IDs; unknown → UNK."""
    return [vocab.get(tok, vocab["[UNK]"]) for tok in sentence.split()]

def decode(ids: list[int]) -> str:
    """Convert IDs back to token strings."""
    return " ".join(id_to_token.get(i, "[UNK]") for i in ids)

# ── Encode ─────────────────────────────────────────────────────────
ids = encode("I love AI")
print(ids)          # [4, 5, 6]

ids_unk = encode("I love pizza")
print(ids_unk)      # [4, 5, 1]  ← pizza → UNK (ID 1)

# ── Decode ─────────────────────────────────────────────────────────
print(decode([4, 5, 6]))    # "I love AI"
print(decode([4, 5, 1]))    # "I love [UNK]"

Part B — HuggingFace tokenizer (GPT-2):

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

# ── Encode ─────────────────────────────────────────────────────────
ids = tokenizer.encode("I love AI")
print(ids)                      # [40, 1842, 9552]  (real GPT-2 IDs)

# ── Decode ─────────────────────────────────────────────────────────
text = tokenizer.decode(ids)
print(text)                     # "I love AI"

# ── Convert to tokens (strings) ────────────────────────────────────
tokens = tokenizer.convert_ids_to_tokens(ids)
print(tokens)                   # ['I', 'Ġlove', 'ĠAI']
# Note: Ġ is GPT-2's representation of a leading space

# ── Get ID of a single token ────────────────────────────────────────
print(tokenizer.convert_tokens_to_ids("love"))   # a specific integer

# ── Special token IDs ──────────────────────────────────────────────
print(tokenizer.eos_token_id)    # 50256 for GPT-2
print(tokenizer.pad_token_id)    # None for GPT-2 (no PAD in GPT-2)

The same surface form gets completely different integer IDs in different tokenizers. Even within one tokenizer, context like a leading space changes the ID.

Try these experiments yourself with a real tokenizer:

Here is exactly where token IDs sit relative to the other representations in an LLM pipeline:

🧠 What does the model know right after tokenization?

Not much yet. After tokenization the text has become discrete token IDs. Those IDs identify tokens, but the integers themselves do not encode meaning in a numeric geometry the transformer can use. Compare the three stages (IDs below are illustrative; real IDs depend on the tokenizer):

• Why not feed IDs directly? cat → 3797 and dog → 8531 does not mean a dog is "more" of anything than a cat. A larger ID is not more important or more similar.
• Same ID, different context. bank has one ID in "The bank approved my loan" and in "We sat on the bank of the river". The transformer builds a different contextual representation for each from the surrounding words.
• Knowledge is not acquired at the embedding step. The embedding matrix and transformer weights were learned in training; the lookup and the layers apply them to your input.

• Model input tensors are integer ID arrays. The first thing any Transformer sees is a torch.LongTensor of shape [batch_size, sequence_length] filled with token IDs.
• Batching requires padding: sentences in a batch have different lengths. All are padded to the same length using PAD IDs so they form a rectangular tensor.
• Attention mask: a companion binary tensor (1 = real token, 0 = PAD) tells the model which IDs are real content versus padding. Without it, the model attends to PAD positions.
• Context length is measured in number of IDs, not words. GPT-4's 128k context means 128,000 token IDs maximum.
• nn.Embedding in PyTorch takes an integer tensor of IDs and returns the corresponding embedding rows — this is the first layer of every language model.

import torch
import torch.nn as nn

# Batch of 2 sentences, padded to length 5 with PAD ID=0
input_ids = torch.LongTensor([
    [4,  5,  6,  0, 0],   # "I love AI [PAD] [PAD]"
    [9, 10,  7,  8, 3],   # "the of ing learning [EOS]"
])

attention_mask = torch.LongTensor([
    [1, 1, 1, 0, 0],   # 1=real token, 0=padding
    [1, 1, 1, 1, 1],
])

# Embedding layer — the first thing the model does with IDs
embedding = nn.Embedding(num_embeddings=50257, embedding_dim=768)
vectors = embedding(input_ids)
print(vectors.shape)   # torch.Size([2, 5, 768])
#   batch=2, seq_len=5, embed_dim=768

🚫 Common Misconceptions to Correct

• Misconception: Token IDs are learned during backpropagation.
  Reality: No. Token IDs are assigned statically before training begins. They are permanent, deterministic indices. The neural network only trains the embedding values that reside at those indices.
• Misconception: Reordering the vocabulary list breaks the model's intelligence.
  Reality: It does, but only because the embedding rows become misaligned. The indices are purely symbolic — if you swap both the vocabulary mapping table and the corresponding rows in the embedding matrix together, the model would still work correctly.
• Misconception: ID 200 is "between" ID 100 and ID 300 in semantic space.
  Reality: No. Semantic space is defined by embedding vectors. The distance between IDs means nothing.

📌 Summary — Everything in Six Bullets

• ID = integer address. A token ID is a non-negative integer that identifies a token by its row position in the embedding matrix. It is not a score, not a frequency, not a magnitude — just an address.
• Embedding matrix lookup. The embedding matrix E has shape [vocab_size, embed_dim]. A token ID N selects row N: E[N] returns the 768-float (or similar) vector that the model actually uses.
• No semantic numeric meaning. Never do arithmetic on IDs. ID 200 is not more meaningful than ID 20. Semantic similarity lives in the embedding vectors, not in the distance between IDs.
• Special token IDs. Reserved IDs (PAD=0, UNK=1, BOS=2, EOS=3 by convention) handle padding, unknowns, and sequence boundaries. Always check your specific tokenizer's special token IDs — conventions differ.
• Padding uses PAD IDs. Batches require rectangular tensors. Shorter sequences are right-padded with PAD IDs. An attention mask (1=real, 0=pad) prevents the model from attending to padding positions.
• IDs are model-specific. BERT ID 500 ≠ GPT-2 ID 500. Token IDs are only meaningful within the tokenizer and model they were created for. Always store the tokenizer name alongside any saved IDs.