Embeddings — My Understanding
Everything on this page is written by me, as I understand it. Not copied from the lessons — just what I took away after reading each subtopic.
Embedding means fetching/mapping the tokenized token number (Token ID) to its pre-trained numerical vector from the vocabulary matrix.
"plant" into an integer ID like 48365. The embedding layer takes that ID and fetches row #48365 from the pre-trained weights table, producing its starting vector representation: [0.82, 0.17, -0.11, -0.94, ...].
Old NLP models used full word embeddings (one vector per dictionary word). Modern LLMs use subword token embeddings (e.g. BPE/Byte-Pair Encoding), allowing the model to handle prefixes, suffixes, typos, and any language without out-of-vocabulary errors.
Token embeddings provide the initial identity of each token piece in the vocabulary. Positional encodings are added to them so the model also knows word order.
The embedding layer is physically stored as a giant weight matrix: [Vocabulary Size × Embedding Dimension] (e.g. 50,257 rows × 768 columns in GPT-2).
- Each Token ID acts as a Row Index.
- Fetching an embedding is a simple table lookup:
Row = Matrix[Token_ID]. - The retrieved vector rows for all tokens in the sentence are packaged together and sent to the next stage: Multi-Head Self-Attention.
human → [height, colour, age, education] and car → [price, colour, model, company]. If every position means something different, how can the Transformer make sense of these numbers at this stage?The gap in my thinking
A real LLM does not use human-designed parameters. An embedding vector is not a database row of properties. It is a learned numerical representation.
A tiny imaginary example (3 dimensions, made-up numbers)
human → [0.8, 0.2, 0.6]car → [0.1, 0.9, 0.4]
Both use the same three dimensions, with different values. Dimension 1 is not necessarily height, and dimension 2 is not necessarily colour. The model learns patterns across all the dimensions together.
No programmer tells the model which dimension means "human" or "car". Training adjusts the embedding values until the whole system is good at predicting language. What the Transformer then does with these numbers, and where the actual meaning comes from, is covered in Self Attention → Context Aware Representation.
Close in embedding space means two vectors point in a similar direction, or sit near each other. The model has learned that those two tokens are used in similar ways, so it treats them as similar in meaning.
How "close" is measured:
- Cosine similarity: the angle between the two vectors. Close means a value near 1.
- Euclidean distance: the straight-line gap between the two points. Close means a small number.
- Dot product: direction and length together. Close means a large number.
For text, cosine similarity is the most common, because it compares direction only and ignores vector length.
Things to remember:
- king and queen end up close because they appear in similar contexts. cat and carburetor end up far apart.
- Close means "used in similar ways", which is not always "means the same". Opposites like hot and cold can end up close.
- A static embedding gives bank one position for both meanings. A contextual embedding moves it nearer to money or nearer to river, depending on the sentence.
- A single dimension has no human-readable meaning. Closeness only makes sense when you compare whole vectors.
📘 Recommended reading on vector math & geometry: The Nature of Code: Vectors ↗ #charan
Your understanding here.
Your understanding here.
Things I'm still unsure about
- Add your doubts here.