Distance Metrics

A distance metric is a rule that takes two points in a vector space and returns a single non-negative number representing how far apart they are. The smaller the number, the closer the points. The metric you choose changes what "close" and "far" mean — two points can be near each other under one metric and far apart under another.

Many core AI tasks are fundamentally nearest-neighbour problems in disguise. Once data is embedded as vectors, the algorithm reduces to: "which vectors are closest to this query?" A single scalar distance lets you rank an entire dataset in one pass.

• Retrieval: Find documents whose embeddings are closest to a query embedding (RAG, semantic search).
• Classification: k-NN classifiers assign labels based on the majority label among the $k$ nearest training examples.
• Clustering: k-means and DBSCAN group points by within-cluster distance.
• Anomaly detection: A point far from all known reference vectors is flagged as an outlier.
• Recommendation: Suggest items whose vectors are close to a user's preference vector.

The best intuition is a city grid. Imagine navigating from one intersection to another:

• Euclidean distance (L2): How far is it as the crow flies? Draw a straight diagonal line between the two points. That length is the Euclidean distance. It does not care about roads.
• Manhattan distance (L1): How far must you walk if you must follow the grid, turning only north/south or east/west? Add up the horizontal and vertical blocks you must travel. No shortcuts.
• Chebyshev distance (L∞): How many moves does a chess king need? The king can move one step in any of 8 directions. The total moves needed equals the largest single-axis gap.

Same two points: A = (1, 2)  →  B = (4, 6)

Euclidean: straight diagonal line, length = √( (4-1)² + (6-2)² ) = √25 = 5
Manhattan: walk right 3 blocks + walk up 4 blocks = 7
Chebyshev: max(|4-1|, |6-2|) = max(3, 4) = 4

All three common metrics belong to the Minkowski family, parameterised by $p$: $d_p(x,y) = \left(\sum_{i=1}^{n} |x_i - y_i|^p \right)^{1/p}$. Setting $p$ to different values gives the familiar named metrics.

All distance computations follow the same two-stage pattern: first compute per-coordinate differences, then aggregate them into a single scalar according to the metric's rule.

Walk through two worked examples in detail so the mechanics become automatic.

2D example: $x = [1, 2]$, $y = [4, 6]$

3D example: $a = [0, 1, 0]$, $b = [3, 1, 4]$

The Mermaid diagram below shows how the same two points A=(1,2) and B=(4,6) produce three different numbers depending on which rule you apply.

First, implement each metric from scratch so you understand every step. Then see the idiomatic NumPy and SciPy equivalents used in production code.

import math
import numpy as np
from scipy.spatial import distance as scipy_dist

# ── From scratch ────────────────────────────────────────────────────────────

def euclidean(x: list[float], y: list[float]) -> float:
    """L2 distance: straight-line separation in n-dimensional space."""
    return math.sqrt(sum((xi - yi) ** 2 for xi, yi in zip(x, y)))

def manhattan(x: list[float], y: list[float]) -> float:
    """L1 distance: sum of absolute coordinate differences."""
    return sum(abs(xi - yi) for xi, yi in zip(x, y))

def chebyshev(x: list[float], y: list[float]) -> float:
    """L-infinity distance: maximum absolute coordinate difference."""
    return max(abs(xi - yi) for xi, yi in zip(x, y))

# ── Test ─────────────────────────────────────────────────────────────────────
a = [1, 2]
b = [4, 6]
print(f"L2  = {euclidean(a, b)}")     # 5.0
print(f"L1  = {manhattan(a, b)}")     # 7
print(f"L∞  = {chebyshev(a, b)}")     # 4

# ── NumPy equivalents ────────────────────────────────────────────────────────
A = np.array([1.0, 2.0])
B = np.array([4.0, 6.0])

print(np.linalg.norm(A - B))          # Euclidean → 5.0
print(np.linalg.norm(A - B, ord=1))   # Manhattan → 7.0
print(np.linalg.norm(A - B, ord=np.inf))  # Chebyshev → 4.0

# ── SciPy: works on any-length arrays ────────────────────────────────────────
print(scipy_dist.euclidean(A, B))     # 5.0
print(scipy_dist.cityblock(A, B))     # 7.0
print(scipy_dist.chebyshev(A, B))     # 4.0

# ── High-dimensional example (typical embedding size) ─────────────────────
rng = np.random.default_rng(42)
v1 = rng.standard_normal(768)
v2 = rng.standard_normal(768)
print(f"768-dim L2 = {np.linalg.norm(v1 - v2):.4f}")
# Expect ~39–41 for random standard normal vectors in 768 dims

Distance and similarity express the same relationship with opposite sign conventions. Many systems (especially vector databases) expose one as their primary interface and let you derive the other.

Distance metrics only produce meaningful results when the underlying vector representation is meaningful. Several failure modes are common:

• Feature scaling destroys L2: If one feature ranges from 0 to 1,000 and another from 0 to 1, the large-scale feature dominates all distances. Standardise or normalise first.
• Magnitude confusion: Raw L2 treats a short vector and a long vector as far apart even if they point in the same direction. Use cosine similarity when direction matters more than magnitude.
• Curse of dimensionality: In very high dimensions, all pairwise distances tend to become similar (concentration of measure). The contrast between the nearest and farthest neighbour shrinks, making nearest-neighbour search less meaningful.
• Metric mismatch: Euclidean distance on one-hot vectors is dominated by which bits happen to be set. It does not reflect semantic similarity the way dense embeddings do.
• Different metrics produce different rankings: The nearest neighbour under L2 may not be the nearest neighbour under L1. Your evaluation metric must match your retrieval metric.

Common misconceptions to tackle:

• "L2 is always better than L1" — No: L1 is more robust to outlier coordinates and preferred for sparse features.
• "Distance tells me semantic meaning directly" — No: it tells you geometric distance in the embedding space. Semantic meaning depends on how well the embedding was trained.
• "All vector databases use the same distance metric" — No: each system has defaults and options; match the metric to how your embeddings were trained.
• "I can mix L2 and cosine similarity rankings in the same retrieval pipeline" — No: they are on incompatible scales and cannot be directly compared or averaged.
• "Normalising vectors makes L2 and cosine equivalent" — Yes, this one is true: when both vectors are unit-length, minimising L2 is equivalent to maximising cosine similarity.

• Distance = scalar separation under a specific rule. It always requires specifying both the vector representation and the metric.
• L2 (Euclidean): Straight-line distance. Penalises large individual deviations quadratically. Most common default in ML.
• L1 (Manhattan): Sum of absolute differences. Robust to outlier coordinates. Good for sparse features.
• L∞ (Chebyshev): Maximum absolute coordinate difference. The single worst dimension determines the result.
• Metric choice matters: Same pair of vectors, different metrics, different rankings. There is no universally "best" metric — it depends on your data and task.
• Feature scaling is non-negotiable with L2/L1: Unscaled features make distance measurements meaningless when features have different ranges.
• High-dimensional warning: Curse of dimensionality can cause distance concentration — all points appear equally far apart, degrading nearest-neighbour quality.
• Distance vs similarity: They express the same relationship with opposite polarity. Cosine similarity is magnitude-invariant; L2 is not.
• Next: Similarity metrics (cosine, dot product) → how vector databases index and search at scale.