Follow the signal through self-attention
Select a step to inspect the numbers
01 / Input token vectors
Each token starts with a small illustrative vector.
Selected token catCurrent step 1 / 5Demo values editable
Attention weight chart
How much the selected token attends to each position.
Weights update when the score controls change. Hover or focus a bar to inspect its value.
Make the numbers yours
Change the illustrative scaled scores. Softmax weights and the weighted output recalculate immediately.
Raise a score to give that token more attention weight, all else equal.
Inspect the Value vectors
These simple 2D Value vectors are for teaching. Their weighted sum changes with the attention weights.
Attention weights determine how much each Value vector contributes to the output.
Educational simplification: real Q, K, and V are learned linear projections of each token representation. The scores here are illustrative scaled scores; real attention commonly uses QKᵀ/√dₖ, then a softmax, followed by a weighted sum of V. Real models use much larger vectors and multiple heads.