Vector Space

A vector space is the mathematical arena in which all of a model's numeric representations live. Every token embedding, every query, every key, every document chunk stored in a retrieval index — all of these are points in some vector space. Understanding the space is what makes all the arithmetic on top of it interpretable.

Once you have a vector space, geometric intuitions give you working tools for reasoning about language:

Imagine a city map where every landmark has precise coordinates. Two cafés placed near each other on the map are physically close in reality. A vector space is similar — except the "city" is built from training data, not geography.

• The coordinates are not hand-crafted. They are learned from billions of examples of how words appear near each other in text.
• "Nearby" in the space means similar according to learned co-occurrence context — not a perfect semantic definition, but a surprisingly useful proxy.
• The dimensions (axes) of the space do not correspond to human-interpretable concepts like "is a mammal" or "has legs". They are abstract features discovered by gradient descent.
• 2D diagrams of vector spaces — including the famous king/queen/man/woman one — are projections. Real models operate in 768 to 8192 dimensions. The 2D picture is a teaching device, not the actual space.

These four terms are often used interchangeably but have distinct roles:

Choosing the wrong metric for a space (e.g. Euclidean distance on unnormalised embeddings of very different magnitudes) gives unreliable results even though the vectors themselves are correct.

In a transformer, the embedding layer is a learned matrix $E \in \mathbb{R}^{V \times d}$ where $V$ is vocabulary size and $d$ is the embedding dimension. Looking up a token is just selecting a row:

Once you have the vectors, the operations inside the model are straightforward linear algebra:

• Addition $x + y$: combine two representations (residual stream, mean pooling)
• Scaling $\alpha x$: amplify or shrink a representation
• Dot product $x \cdot y = \sum_i x_i y_i$: measure alignment (attention scores)
• Euclidean distance $\|x - y\|_2$: straight-line gap between two points (clustering, k-NN retrieval)

Four operations cover most of what happens inside a model's use of vector space:

Use these invented 3D vectors as a learning exercise. The specific numbers don't correspond to any real embedding — they are chosen to illustrate how operations reveal geometric relationships.

Toy vectors (3D):

cat   = [0.9, 0.1, 0.8]
dog   = [0.9, 0.3, 0.9]
tiger = [0.9, 0.8, 0.0]
car   = [0.0, 0.6, 0.0]

The numbers show: cat–dog are more aligned than cat–tiger, and cat–car are nearly orthogonal. That's the geometry encoding learned relationships — without any explicit programming of "animal similarity".

A toy 2D projection of the vectors above. Remember: this is a visualization device only — the actual space has far more dimensions.

Interactive similarity demo

Enter two comma-separated vectors below (same number of dimensions). The demo computes dot product, magnitudes, and cosine similarity.

A minimal implementation of the three core vector-space operations. This is pure Python with no dependencies — focus is on understanding the mechanics before reaching for NumPy or a vector library.

def dot(a: list[float], b: list[float]) -> float:
    """Dot product: sum of element-wise products.
    Measures how much two vectors 'point in the same direction'.
    Large positive → strongly aligned. Near zero → nearly orthogonal.
    """
    assert len(a) == len(b), "Vectors must have the same dimension"
    return sum(ai * bi for ai, bi in zip(a, b))


def magnitude(a: list[float]) -> float:
    """Euclidean norm (length) of a vector.
    Used to normalise vectors and to compute Euclidean distance.
    """
    return sum(ai ** 2 for ai in a) ** 0.5


def cosine_similarity(a: list[float], b: list[float]) -> float:
    """Cosine similarity: dot product divided by both magnitudes.
    Range: [-1, 1].  1 = identical direction.  0 = orthogonal.  -1 = opposite.
    Insensitive to vector length — only measures angle.
    """
    mag_a = magnitude(a)
    mag_b = magnitude(b)
    if mag_a == 0 or mag_b == 0:
        return 0.0
    return dot(a, b) / (mag_a * mag_b)


# ── Toy examples ─────────────────────────────────────────────────────────────
cat   = [0.9, 0.1, 0.8]
dog   = [0.9, 0.3, 0.9]
car   = [0.0, 0.6, 0.0]

print(f"dot(cat, dog)            = {dot(cat, dog):.4f}")        # 1.5600
print(f"dot(cat, car)            = {dot(cat, car):.4f}")        # 0.0600
print(f"cosine(cat, dog)         = {cosine_similarity(cat, dog):.4f}")  # ≈ 0.9864
print(f"cosine(cat, car)         = {cosine_similarity(cat, car):.4f}")  # ≈ 0.0534

# ── With NumPy (production style) ─────────────────────────────────────────────
import numpy as np
cat_np = np.array([0.9, 0.1, 0.8])
dog_np = np.array([0.9, 0.3, 0.9])

dot_prod   = np.dot(cat_np, dog_np)
cos_sim    = np.dot(cat_np, dog_np) / (np.linalg.norm(cat_np) * np.linalg.norm(dog_np))
print(f"NumPy cosine(cat, dog)   = {cos_sim:.4f}")

• Geometry is only as good as the representation: The distances and directions in the space reflect the training objective, not an absolute semantic truth. A model trained on biased data produces a biased space.
• High similarity ≠ equivalent meaning: Words that appear in similar contexts get close vectors, but context similarity is not the same as meaning equivalence. "Not bad" and "good" may both appear near "acceptable" for contextual reasons.
• Static embeddings and polysemy: A word like "bank" has multiple senses. In a static embedding (word2vec, GloVe) it gets one averaged vector that conflates all senses. Contextual models (BERT, LLaMA) help — but the representational tradeoff remains.
• Individual dimensions are not interpretable: Unlike one-hot encoding where each dimension has a defined meaning, learned embedding dimensions are distributed features. You cannot look at dimension 42 and know what it represents.
• Curse of dimensionality: In very high-dimensional spaces, distances between random points concentrate around a single value — almost every pair of points appears "equally far apart". This makes intuitions from 2D or 3D geometry unreliable.

Common misconceptions to address:

• "The vector space is a database of word meanings" — No: it is a geometric environment whose structure reflects training co-occurrence, not a hand-labelled dictionary.
• "Nearby vectors are synonyms" — No: they share contextual patterns, not necessarily semantic equivalence.
• "I can read the meaning of each dimension axis" — No: dimensions are abstract distributed features, not interpretable slots like "is an animal".
• "Distance always gives the ranking I want" — No: the choice of metric (cosine vs Euclidean vs dot) matters enormously, and the right choice depends on how the embeddings were trained.
• "All vector spaces are the same" — No: different models, objectives, and training corpora produce very different spaces even at the same dimensionality.

• Vector space: A high-dimensional coordinate system where every representation is a point. Defined by coordinates (vectors) and allowed operations (add, scale, dot product, distance).
• Vectors: Ordered lists of real numbers that locate a point. Learned from data via gradient descent; individual dimensions are not interpretable.
• Geometry is the mechanism: Closeness encodes contextual similarity, directions encode relational structure, dot products drive attention, cosine similarity drives retrieval.
• Core metrics: Dot product (attention), cosine similarity (semantic comparison, RAG), Euclidean distance (clustering, k-NN). Choose based on how the space was trained.
• Limits: The geometry reflects training data, not absolute meaning. High similarity ≠ equivalence. Dimensions are not interpretable. High-dimensional spaces behave counter-intuitively.
• Connects to: Embeddings (what lives in the space), Attention (QK dot products), Similarity (how to compare), Distance (how to measure gaps), RAG (retrieval as nearest-neighbour search).