🏗️ How Embeddings Are Learned

1. Starting point: random vectors

Imagine the model begins with:

cat → [0.03, -0.11, 0.07, ...]
dog → [-0.08, 0.14, -0.02, ...]
car → [0.09, 0.01, -0.15, ...]

At this stage, these numbers do not mean much.

They are just initial parameters.

2. What training is trying to do

The model is trained on prediction tasks.

Examples:

• predict the next token
• predict a masked token
• predict surrounding words
• improve downstream task performance

When the model makes a prediction error, training adjusts many parameters, including embeddings.

So over time:

• useful token relationships get reinforced
• unhelpful patterns get reduced
• embeddings become better for the model’s objective

3. Simple intuition

Suppose the model repeatedly sees contexts like:

The cat sat on the mat
The dog sat on the rug

Then cat and dog may start getting more similar embeddings because they appear in similar contexts.

This is the core intuition behind learned embeddings:

words used in similar contexts
→ often get similar representations

This idea is closely related to the distributional hypothesis.

4. Embedding table view

A practical way to think about it:

• vocabulary size = number of rows
• embedding dimension = number of columns

Example:

Embedding matrix:
50,000 × 768

Each row corresponds to one token.

Training updates the values in these rows.

So learning embeddings means learning the embedding matrix.

5. What actually updates them?

During training:

1. input tokens are converted to embeddings
2. the model makes predictions
3. loss is computed
4. backpropagation calculates gradients
5. optimizer updates parameters

Embeddings are part of those parameters.

So they improve through the same learning loop as the rest of the model.

6. Important point: embeddings are task-shaped

Embeddings are not learned in isolation.

They are shaped by the training objective.

That means:

• embeddings learned for language modeling may differ from embeddings learned for retrieval
• embeddings learned in one model may not behave exactly like embeddings in another
• the same token can be useful in different ways depending on the model’s goal

So embeddings are not universal truth vectors.

They are learned representations optimized for a training setup.

7. Why this matters

This explains why embeddings can capture:

• similarity
• analogy-like patterns
• topic relationships
• syntactic behavior
• semantic usefulness

Not because someone manually encoded those ideas, but because the model kept adjusting vectors to reduce prediction error.

8. Common misunderstanding

Do not think:

the model first learns English rules,
then separately writes embeddings

That is not how it works.

Instead, embeddings and the rest of the model improve together during training.

They co-evolve.