Self Attention — My Understanding
Everything on this page is written by me, as I understand it. Not copied from the lessons — just what I took away after reading each subtopic.
Your understanding here.
[height, colour, age, education]), how can the Transformer make sense of these numbers?The correction
The vector is not a row of human-designed properties. It is a learned numerical representation, so there is no per-position meaning for the Transformer to decode. In a tiny made-up example, human → [0.8, 0.2, 0.6] and car → [0.1, 0.9, 0.4] use the same dimensions with different values.
What the Transformer does with them
Training on sentences like "The human drives the car" teaches the model numerical transformations that help it predict the missing or next word. For "The human drives the ...", it learns to use information from human and drives to predict car.
- Input vector:
human → [0.8, 0.2, 0.6] - Learned weights turn it into Q (what to look for), K (what matches) and V (information to share).
- Attention compares Q with K and combines V.
- Result: a contextualized vector, where "human" now carries relevant context.
Training adjusts the embedding values and the Transformer's weights together, so nobody needs to say which dimension means what.
Try it: vector meaning visualiser (flip between untrained and trained weights).
Worked by hand: numerical walkthrough (the embedding for "cat" → Q·K scores → softmax → weighted sum of V → context-aware vector).
🎓 The Student Exam Analogy: Training vs. Inference (Generalization) #charan
1. Training Phase (Studying for the Exam): A student practices math problems like 5 + 2 = 7. While studying, he makes errors, corrects himself, and internalizes the underlying logic of addition (this is backpropagation adjusting the model's weights).
2. Inference Phase (The Exam Hall): In the exam, he encounters a brand new question he never saw in his textbook: 9 + 7 = ?. Because his brain mastered the underlying logic during study, he accurately solves it as 16 without needing to have memorized that exact combination.
That is exactly what happens during Inference: the Transformer's trained weights are frozen, but because it learned generalizable linguistic and mathematical transformations during training, it can assemble accurate Context Vectors for brand-new sentences it has never seen before!
The one sentence I want you to remember #charan
That's how the numbers become useful. Not because each number has a fixed human-readable meaning, but because the entire learned system works together.
How Scaling / Temperature works:
Before passing scores into Softmax, we divide them by a scaling factor $T$ (in attention, $T = \sqrt{d_k}$, the square root of the head dimension):
- High Temperature ($T > 1$): Flattens the scores → attention is distributed broadly across many words (softer, more creative spread).
- Low Temperature ($T \to 0$): Amplifies score differences → the top-matching word dominates with ~100% weight (sharper, deterministic focus).
- Standard Attention Scaling ($\sqrt{d_k}$): Stabilizes training so softmax outputs healthy gradients.
[-3.2, 0.0, 8.4]). How does the model turn these into valid weights?
Why Softmax is essential #charan
We cannot blend vectors using negative numbers or arbitrary magnitudes. Attention weights must act like percentages that add up to exactly 1.0 (100%).
Weight_i = e^(Score_i / T) / Σ (e^(Score_j / T))
1. Exponentials (e^x): Turn every negative or zero score into a positive number.
2. Normalization (divide by sum): Guarantees all weights sum to exactly 1.0 (100%).
These softmax weights are then multiplied by each token's Value (V) vector to assemble the updated context vector!
Self-attention helps the model learn how the tokens relate to each other.
Example: in "Deposit money in the bank", the words deposit and money pull "bank" toward the finance meaning. In "Sit on the river bank", the words sit and river pull it toward the land-beside-a-river meaning. The word "bank" is the same, but the context decides which meaning wins.
Try it: attention simulator (click "bank" and compare the two presets).
Your understanding here.
Your understanding here.
Things I'm still unsure about
- Add your doubts here.