🧠 Embeddings

1. What embeddings solve

Embeddings solve the main weakness of token IDs and one-hot vectors:

That means the model can learn that:

A dense vector means most positions contain useful values.

Example:

[0.12, -0.44, 0.91, 0.08]

This is dense.

Compare with one-hot:

[0, 0, 1, 0]

This is sparse.

Dense vectors are better because they:

At a simple level, each token ID points to a row in an embedding table.

Example:

token ID 4821 → row 4821 in embedding matrix

So if the vocabulary size is 50,000 and embedding size is 768, then the embedding table is:

50,000 × 768

Each row is the vector for one token.

This table starts random and gets improved during training.

An embedding is not a dictionary definition.

It is not a human-readable explanation.

It is a learned numeric representation that becomes useful because of how the model trains.

So the model does not store:

cat = small furry animal

Instead it learns a vector that helps it behave correctly when cat appears in text.

Embeddings are the bridge between:

Without embeddings, the model cannot move from token identity to usable meaning patterns.

This is why embeddings are foundational for:

At first, you can think of embeddings as one vector per token.

Example:

bank → one vector

But later, modern transformers make this more powerful:

bank in "river bank"      → one contextual representation
bank in "bank account"    → another contextual representation

That later idea is covered in: