Vector Dimensions

A dimension is a single coordinate slot in a vector. A 3D vector has three slots: [x, y, z]. A 768-dimensional embedding has 768 slots: [d₀, d₁, …, d₇₆₇]. Every slot holds one floating-point number.

2D  →  [ 0.2,  0.8 ]
3D  →  [ 0.2,  0.8, -0.4 ]
5D  →  [ 0.2,  0.8, -0.4,  0.1,  0.6 ]
768D→  [ 0.21, -0.07, 0.54, 0.11, …, 0.33 ]   (768 values)

Concrete example: the vector for “cat”

Suppose the token ID for cat is 1. Its row in the embedding matrix is (illustrative values):

Token ID = 1   → which row to read
Vector for cat = [  0.80,  −0.30,   0.40,   0.15 ]
                   dim 1   dim 2   dim 3   dim 4

4 numbers = a 4-dimensional vector

Each dimension is one position in the vector. Together, the numbers form the token's learned numerical representation. The token ID is not this vector; it only tells the model which vector to retrieve.

Try it: vector dimension explorer

Drag the slider to change how many numbers describe one token. Bars above the line are positive values, bars below are negative.

Every extra dimension added to your embedding is an extra degree of freedom — the model can encode one more independent aspect of meaning. But that freedom comes at a price paid in every operation that touches the vector.

If you double d from 384 to 768, you double the memory for every stored embedding, and every dot product in every attention head takes twice as many multiply-accumulate operations. The embedding matrix itself grows by a factor of two as well.

How big are vectors in real models?

These are example sizes; actual models vary. With 4,096-dimensional embeddings, each token ID retrieves a vector containing 4,096 numbers.

Imagine plotting words on a 2D graph where one axis loosely captures "animal-ness" and another captures "size". You could put cat at (0.8, 0.3) and elephant at (0.7, 0.95). That works for two features. But real language has thousands of interacting properties.

Real embeddings don't assign clean human-readable labels to their dimensions. They are learned: gradient descent arranges the numbers so that vectors which should behave similarly in downstream tasks end up geometrically close. What ends up encoded in dimension 237 is an entangled mixture of many features — not a single interpretable concept.

A human analogy

Describe a person to someone who has never seen them, using 4 measurements:

• Person → the thing being represented (here, the token).
• Measurements → individual dimensions.
• List of measurements → the vector.
• Number of measurements → the dimensionality.

Describing a person with hundreds of measurements captures more detail than four, and in the same way a model may use 4,096 dimensions per token. More dimensions alone do not guarantee a better model; architecture, training and the learned parameters matter too.

Three numbers appear together constantly when reading NLP papers and tensor shapes. They describe different axes of the same tensor and must not be confused.

Hidden-state tensor shape in BERT-base:
  [B=8,  T=128,  H=768]
   ^      ^       ^
   batch  seq     embedding dim
   (how   (how    (how wide
   many   long)   each vector is)

The embedding layer is a matrix E of shape [V, d] where V is vocabulary size and d is embedding dimension. Given a token ID i, the embedding for that token is simply row i of E: a vector of length d.

Notice that every Transformer layer preserves the hidden dimension d. Attention projects queries, keys, and values to subspaces of size d_k (often d / num_heads), then projects back to d before adding the residual. The width of the representation stays constant through the stack.

The rules of vector algebra are identical whether you have 2 dimensions or 768. Here are concrete vectors at four different widths, all using the same dot-product formula.

To visualize a set of 768D embeddings, researchers project them down to 2D or 3D using dimensionality reduction techniques. The most common are PCA, t-SNE, and UMAP.

The most common first step when working with embeddings is simply inspecting shapes to confirm that dimension, sequence length, and batch size are in the right places.

# ── Plain Python ───────────────────────────────────────────
tokens = 4          # sequence length (4 tokens)
embedding_dim = 6   # dimension (6 coordinates per token)

# A manual embedding table: list-of-lists
embedding_table = [
    [ 0.21, -0.07,  0.54,  0.11, -0.33,  0.08],   # token 0
    [-0.15,  0.42,  0.03, -0.28,  0.67,  0.21],   # token 1
    [ 0.88, -0.11, -0.09,  0.55,  0.04, -0.19],   # token 2
    [ 0.10,  0.30, -0.74,  0.02,  0.51,  0.44],   # token 3
]

num_tokens = len(embedding_table)      # → 4  (sequence length)
dim_per_token = len(embedding_table[0])  # → 6  (embedding dim)
print(f"Tokens: {num_tokens}, Dim: {dim_per_token}")

# ── NumPy ───────────────────────────────────────────────────
import numpy as np

E = np.array(embedding_table)
print(E.shape)       # (4, 6)  →  [seq_len, embedding_dim]
print(E[0].shape)    # (6,)    →  one token vector

# Batch of sequences
batch = np.random.randn(8, 128, 768)   # [batch, seq, dim]
print(batch.shape)   # (8, 128, 768)

# ── PyTorch ────────────────────────────────────────────────
import torch
import torch.nn as nn

vocab_size = 30522
d_model = 768

emb = nn.Embedding(vocab_size, d_model)
print(emb.weight.shape)   # torch.Size([30522, 768])

token_ids = torch.tensor([[101, 2054, 2003, 102]])  # shape [1, 4]
out = emb(token_ids)
print(out.shape)     # torch.Size([1, 4, 768])
#                                  ^  ^    ^
#                               batch seq  dim

• More dimensions ≠ better representations. A poorly-trained 768D model will underperform a well-trained 128D model. Quality of training data and objectives matters far more than dimension count.
• Individual dimensions are not interpretable. You cannot read "dimension 237 = formality". Each dimension is a learned mixture. Probing classifiers can sometimes find linear directions, but the raw coordinates are not meaningful.
• Visualization requires lossy projection. PCA, t-SNE, and UMAP all discard or distort structure. A 2D plot of 768D data should be treated as a rough sketch, not ground truth.
• Curse of dimensionality. In very high-dimensional spaces, all pairwise distances tend to become similar, which can make nearest-neighbour search less discriminative. Approximate nearest-neighbour methods (HNSW, IVF) are needed at scale.

Dimension is the width of a single vector — the number of coordinate slots. It determines representational capacity and is directly proportional to memory and compute cost.

• A vector in \(\mathbb{R}^d\) has exactly \(d\) coordinates — each is one dimension.
• Embedding matrix shape: [V, d]. Hidden-state tensor shape: [B, T, d].
• Dot product and cosine similarity both require vectors of the same dimension — mixing models with different d silently breaks retrieval.
• Individual dimensions are not interpretable — only the vector as a whole is meaningful.
• Use dimensionality reduction (PCA / t-SNE / UMAP) only for visualization; never for measuring actual similarity.
• Common real-world sizes: 128 (tiny) → 384 (compact) → 768 (standard) → 1024 (large) → 4096 (generative LLMs).