Why Text Becomes Numbers

Every modern language model — BERT, GPT-4, LLaMA, T5 — shares one non-negotiable requirement before a single weight update can happen: text must be converted into numbers. Neural networks are mathematical machines. Every operation inside them — matrix multiplications, dot products, softmax activations, gradient descent — is defined only over real numbers. They have no concept of the string "cat", the word "democracy", or the sentence "the sky is blue".

This page answers the most fundamental question in NLP: why does text become numbers, and how does that conversion happen at each level — from character, to token, to integer ID, to a dense floating-point vector?

The requirement for numeric representation is not a limitation to work around — it is the foundation on which every learning algorithm is built. Here are the four pillars:

• Compute (matrix operations): Attention mechanisms, feed-forward layers, and embeddings all require matrix multiplication. You cannot multiply strings. A Transformer's self-attention is fundamentally $\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$ — every symbol here is a real-valued matrix.
• Learn (gradient descent): Training adjusts parameters by computing the gradient of a loss function with respect to those parameters. Gradients require derivatives, and derivatives require continuous numeric values. Discrete symbols like words have no natural gradient.
• Compare (dot products and distances): The model must assess how similar or related two pieces of text are. That comparison is geometric — cosine similarity, Euclidean distance — and geometry requires coordinates, not strings.
• Generalise (dense representations): Sparse representations (one word = one dimension) generalise poorly. Dense vectors allow the model to place similar concepts near each other in a continuous space, sharing learned properties across related tokens.

Imagine a global airport database. Every airport in the world has a 3-letter IATA code: JFK, LHR, NRT. Those codes are compact identifiers — analogous to token IDs. They tell the system which airport to look up, but the code itself says nothing about the airport's location, runway count, altitude, or passenger volume.

Behind each code is a full record: latitude, longitude, number of terminals, elevation, regional hub status, connection density, average delay. That rich record is the embedding vector. The code is an index; the record is the learned description.

Airport analogy:

  "cat"  →  token ID 4127  →  embedding [ 0.12, -0.41,  0.77,  0.23, -0.18, … ] (768 dims in BERT)
  "dog"  →  token ID 5863  →  embedding [ 0.09, -0.38,  0.81,  0.21, -0.15, … ]
  "car"  →  token ID 2912  →  embedding [-0.44,  0.22, -0.31, -0.52,  0.63, … ]

  cat and dog vectors are close  →  model learned they share properties (animals, domestic, etc.)
  car vector is far from both    →  model learned it belongs to a different region of meaning

This two-level design is deliberate. Token IDs are integers because integer indexing into an array is computationally free. The meaning-carrying work is entirely delegated to the embedding matrix, which is trained to place semantically similar tokens near each other in the vector space.

Moving from raw text to model-ready input involves four distinct levels of abstraction, each solving a different problem:

In practice, modern neural networks skip the explicit one-hot step and directly implement the one-hot → dense multiplication as an embedding lookup (selecting a matrix row), which is computationally equivalent but far more efficient for large vocabularies.

Here is the concrete pipeline for turning a sentence into model-ready vectors:

The embedding matrix $E$ starts with random values and is updated during training via backpropagation. By the end of training, its rows encode the full semantic and syntactic knowledge the model has extracted from the training corpus. The matrix is essentially the model's entire vocabulary of learned concepts, stored as geometry in a $d$-dimensional space.

Let's be precise about the mathematics. The embedding layer defines a function:

$$E : \{0, 1, \ldots, V-1\} \rightarrow \mathbb{R}^d$$

where $V$ is the vocabulary size and $d$ is the embedding dimension. Concretely, $E$ is implemented as a matrix $\mathbf{E} \in \mathbb{R}^{V \times d}$, and the embedding of token $t_i$ is the $t_i$-th row:

$$\mathbf{x}_i = \mathbf{E}[t_i] \in \mathbb{R}^d$$

The key transition this formalises is discrete → continuous: token IDs are elements of a finite discrete set $\{0, \ldots, V-1\}$; embedding vectors are points in the continuous space $\mathbb{R}^d$. This is what makes gradient descent possible — you can take derivatives with respect to the entries of $\mathbf{E}$.

Use this small example to see all four representation levels side by side. Vocabulary of 5 tokens: the, cat, sat, dog, ran.

What can you observe from this toy table?

• ID 1 vs ID 3: The integers 1 and 3 say nothing about their relationship. But the dense vectors for cat and dog are very similar — their first dimension is high and positive for both, and their third dimension is negative for both. The model has learned they share properties.
• sat vs ran: Both have high values in dimension 2 and 3. They are both verb-like. Their IDs (2 and 4) are completely uninformative about this.
• the: Very small values across all dimensions — as a function word, it has weak standalone semantics and sits near the origin in this toy space.
• One-hot is flat: Every token is equidistant from every other token in one-hot space. The cosine similarity between any two distinct one-hot vectors is exactly 0. Dense vectors break this symmetry by placing related tokens closer together.

The diagram below shows the complete transformation from raw text to the dense vectors that enter the first layer of a Transformer. Each arrow represents one stage of processing:

The critical bridge is the embedding matrix lookup (third arrow). Before that step, we are in the discrete world of integers. After that step, we are in the continuous world of real-valued vectors where all of the model's learned geometry lives.

The positional embedding added in the final step encodes where in the sequence each token appears — without it, the model would treat "cat sat" and "sat cat" identically.

This implementation shows every stage of the pipeline using only built-in Python — no NumPy, no PyTorch, no ML library. The goal is to make the mechanics transparent.

"""
Text-to-vector pipeline in plain Python.
Shows each stage: token → ID → (one-hot) → dense vector.
Does NOT learn or update the embedding table — that happens during training.
"""

# ── Vocabulary (token → integer ID) ─────────────────────────────────────────
vocab = {
    "the": 0,
    "cat": 1,
    "sat": 2,
    "dog": 3,
    "ran": 4,
}

# ── Toy embedding table (pre-trained / random initialisation) ────────────────
# Shape: [vocab_size, embedding_dim] = [5, 3]
# In real models this is learned from data. Here we use hand-crafted values
# that illustrate semantic clustering.
embedding_table = [
    [ 0.03, -0.02,  0.01],   # ID 0  → "the"   (function word, near origin)
    [ 0.85,  0.62, -0.44],   # ID 1  → "cat"   (animal, domestic)
    [-0.12,  0.78,  0.91],   # ID 2  → "sat"   (verb, stationary)
    [ 0.82,  0.59, -0.41],   # ID 3  → "dog"   (animal, domestic — close to cat)
    [-0.09,  0.77,  0.88],   # ID 4  → "ran"   (verb, motion — close to sat)
]

def text_to_ids(text: str) -> list[int]:
    """Step 1 & 2: tokenize (split on whitespace) and map to token IDs."""
    tokens = text.lower().split()
    ids = []
    for token in tokens:
        if token not in vocab:
            raise ValueError(f"Token '{token}' not in vocabulary")
        ids.append(vocab[token])
    return ids

def ids_to_vectors(ids: list[int]) -> list[list[float]]:
    """Step 3: embedding lookup — select the row for each ID."""
    # Equivalent to: one_hot @ embedding_table  (but O(n) not O(n*V))
    return [embedding_table[i] for i in ids]

# ── Pipeline ─────────────────────────────────────────────────────────────────
text = "the cat sat"

token_ids = text_to_ids(text)
print("Token IDs:", token_ids)           # [0, 1, 2]

vectors = ids_to_vectors(token_ids)
for tok, tid, vec in zip(text.split(), token_ids, vectors):
    print(f"  '{tok}' (ID {tid}) → {vec}")

# Output:
#   'the' (ID 0) → [0.03, -0.02, 0.01]
#   'cat' (ID 1) → [0.85, 0.62, -0.44]
#   'sat' (ID 2) → [-0.12, 0.78, 0.91]

# ── What this code does NOT do ───────────────────────────────────────────────
# 1. Learn vectors from data (no backprop, no gradient updates)
# 2. Handle subword tokenization (BPE / WordPiece / SentencePiece)
# 3. Add positional embeddings
# 4. Handle OOV tokens gracefully (would need [UNK] fallback)
# 5. Run in a neural network context (PyTorch: nn.Embedding does all of this)

Type a word from the toy vocabulary (the, cat, sat, dog, ran) and see the token ID and embedding vector at each stage.

Each representation solves a different problem. Understanding when to use which is essential for debugging and designing NLP systems.

The text-to-vector conversion is not just an entry-point detail — it shapes how every downstream NLP system is designed and debugged.

Converting text to vectors is powerful, but several important properties of language are either lost or distorted in this process:

• Numbers ≠ understanding: A vector that places "cat" near "dog" does not mean the model "understands" what cats are. It means those tokens appeared in similar linguistic contexts during training. Correlation in context, not semantic comprehension.
• Token ID is not a semantic score: Nothing about the integer 4127 says anything about the meaning of "cat". Models that inadvertently treat token IDs as meaningful numbers will learn incorrect relationships. This is a real bug in naive implementations.
• One-hot vectors encode no relationships: Before training, every token is equidistant from every other. The model must learn all semantic structure from scratch. Pre-trained embeddings (word2vec, GloVe, or embeddings from large LMs) warm-start this learning.
• Embeddings inherit training biases: If the training corpus contains societal biases, those biases are encoded into the geometric structure of the embedding space. The embedding of "nurse" may be closer to "woman" than "doctor" in a biased corpus.
• Similar vectors are evidence, not proof: High cosine similarity between two token vectors suggests they appear in similar contexts. It does not guarantee they are synonyms or that they are interchangeable in all contexts.

Common misconceptions to tackle:

• "Token IDs encode meaning" — No. IDs are just row indices into the embedding table. The number 4127 has no semantic content. Only the vector at that row does.
• "One-hot vectors encode relationships" — No. Every token is equally distant from every other in one-hot space. The cosine similarity of any two distinct one-hot vectors is exactly 0.
• "I can use token IDs directly as model input features" — No (almost never). Feeding IDs directly to a linear layer treats them as numeric measurements, introducing false ordinal relationships. Always use the embedding lookup.
• "The vector IS the word" — Not quite. The vector is a learned approximation of the word's distributional properties in the training corpus. It captures what contexts the word appeared in, not what the word "is".

Here is everything this page covered, condensed to the essential points:

• Text → numbers is mandatory: Neural networks only operate on real-valued matrices. Strings, words, and tokens must be converted before any computation can happen.
• Four reasons: Compute (matrix ops), learn (gradient descent), compare (geometry), generalise (dense representation).
• Four levels: Token (text) → Token ID (integer) → One-hot (sparse binary) → Dense embedding (learned real-valued vector).
• Token ID is an address, not a score: It tells the model which row of the embedding matrix to retrieve. Adjacent IDs do not imply semantic similarity.
• Dense vectors are where meaning lives: The embedding matrix $\mathbf{E} \in \mathbb{R}^{V \times d}$ is learned from data. Its rows encode the geometric structure of language as distilled from the training corpus.
• The conversion is a bridge: Discrete → continuous. Before the bridge: combinatorial symbol space. After the bridge: differentiable geometric space.