∿
AI JOURNEY / LEARNING LABS
CORE · REPRESENTATIONS 01 / Embeddings
CORE CONCEPT · FROM SYMBOLS TO NUMBERS

Words become
vectors.

An embedding is a learned list of numbers associated with a token. It gives a model a numerical starting point for representing language—so later layers can transform those representations using context.

Explore the live lab ↓See the learning path
Representation engineIllustrative · 6 dimensions
TOKEN“cat”text symbol TOKEN ID42 EMBEDDING TABLE…414243… VECTOR0.24−0.810.37… ID selects a row. The row’s values are the embedding.The ID itself does not contain meaning.
LookupNot generation
LearnedDuring training
DenseMany dimensions
FOLLOW THE TRANSFORMATION

One token. A sequence of representations.

Select a stage to move the explanation, formula, and highlighted signal together.

STAGE 01 · INPUT

Text is split into tokens.

A tokenizer maps a string into token pieces. A token might be a word, part of a word, punctuation, or another learned unit.

WHAT CHANGES HERE?
"cat" → tokenizer → token piece "cat"
No embedding values have been looked up yet.
LIVE EMBEDDING EXPLORER

Inspect the row. Change the representation.

Choose a token, inspect its illustrative ID, and see how its vector coordinates change when you alter the values or add positional information.

Embedding table lookup

Select a token to inspect its row
Live values
Selected token ID42“cat”
row address
EMBEDDING MATRIX EE[42]
dim
0.240
−0.810
0.370
0.120
−0.440
0.680
Six visible dimensions for learning. Real embedding vectors often have hundreds or thousands of dimensions; this display is deliberately reduced.

Token embedding · e

After position · e + p

Manipulate the numbers

The bars and output vector update immediately
Experiment

Change the selected embedding's coordinates. This is a teaching sandbox—not a live model weight editor.

Controls how strongly the example position vector is added.
Different positions use different illustrative position vectors.
The token ID is a lookup address. The values in its row—not the integer ID—are the embedding.
BUILD THE RIGHT MENTAL MODEL

What embeddings do—and don’t do.

01

An embedding is a vector

A list of learned numerical coordinates associated with a token. Each coordinate contributes to a representation; individual dimensions rarely have simple human-readable meanings.

02

The ID is not the meaning

Token IDs are categorical identifiers. ID 156 is not more meaningful than ID 42; the embedding table maps each ID to its learned vector.

03

Similarity is learned, not guaranteed by the ID

Training shapes the geometry of vectors. Similarity can be measured with cosine similarity or distance, but this page’s sample numbers are illustrative.

04

Contextual meaning comes later

The input embedding is a starting representation. Transformer layers use context to produce contextualized hidden states that depend on the surrounding tokens.