01 · INPUT
Tokens in a sequence
The sentence is represented as token pieces, each with a position and an initial vector.
Topic 02 · Understand Context
Attention is
selective focus.
A language model needs more than the words themselves. It needs to decide which other words matter for the token it is processing right now. Attention makes that focus dynamic.
LIVE ATTENTION EXPLORERSentence: “Adavitha is playing with Charan in the park.” Select a word to inspect the context it may use.● INTERACTIVE
ATTENTION LINKS · CONCEPTUAL EXAMPLE
playing is the current focus. Attention may connect it to Adavitha, with, and Charan.
SELECT ANY WORD TO CHANGE FOCUSIllustrative relationship · not model scores
LIVE CONCEPT MODEL · WORD BY WORD
One sentence, different attention relationships
Sentence: “Adavitha is playing with Charan in the park.” Choose a word in the sentence above or in this table. The selected row and its attention words will be highlighted together.
| Word | Attention words | Why? |
|---|---|---|
| playing, Charan | Who is playing, and with whom? | |
| playing | Connects the subject to the action. | |
| Adavitha, with, Charan | Identifies the action and participants. | |
| playing, Charan | Connects the action to the person joining in. | |
| Adavitha, playing | Identifies the playmate. | |
| playing, park | Connects the action to the location. | |
| park | Helps specify the place. | |
| playing, in | Identifies where the action happens. |
Specific attention example: When processing “playing,” the model may pay strong attention to “Adavitha” and “Charan” to understand who is playing and with whom. The word “with” also helps express their relationship in this sentence.
Teaching illustration only: these links explain plausible context relationships. They are not attention weights measured from a trained model.
INTERACTIVE LAB · THREE.JS + D3 + GSAP
See attention in 3D.
Explore the orbital token field from Attention-Lab-3D: glowing token spheres, floating labels, a central geometric core, orbital rings, and animated weighted connections. The matrix, focus token, and explanation stay synchronized.
01 / 3D ATTENTION MAPSCENE READY
Loading the 3D scene…
Three.js is initializing. An internet connection is needed for the visualization libraries.
Three.js is initializing. An internet connection is needed for the visualization libraries.
CURRENT QUERY TOKEN
playing
The current token can draw useful context from the subject, the relationship word, and the other participant.
02 / ATTENTION MATRIX
LOWHIGHCell = illustrative weight
HOW TO READ THIS
Rows ask. Columns contribute.
Each row is the focus token (query). Each column is a possible source of information (key/value position). Brighter cells mean a larger illustrative weight for this teaching example. Select any cell to inspect that relationship in 3D.
Teaching model only: weights are hand-authored to demonstrate the idea. They are not measured from a trained model, and the “heads” are conceptual lenses—not real learned attention heads.
01 / THE BIG PICTURE
Where attention fits in the journey
Attention is not the whole language model. It is a mechanism inside the Transformer that helps each token build a representation informed by other tokens. Follow the flow below; later subtopics unpack each step.
02 · MATCH
Compare relevance
Query and key vectors produce scores that indicate which tokens are relevant to the current token.
03 · WEIGHT
Turn scores into weights
Softmax converts scores into normalized attention weights for the chosen focus token.
04 · CONTEXT
Mix useful information
The weights combine value vectors to create a context-aware representation.
02 / CORE INTUITION
One sentence. Different questions. Different focus.
Attention is contextual: the useful evidence for one word may differ from the evidence needed for another. This is the key idea to understand before diving into Q, K, V, scores, and matrices.
Try it above
Choose a token in the live example. The highlighted token is the one whose context we are imagining. The other highlighted words represent possible sources of useful information.
contextᵢ = Σⱼ weightᵢⱼ × valueⱼ
The equation says: for the current token i, take information from tokens j, scale each contribution by its attention weight, then add the contributions together.
What to notice
01
Focus is local to a query
The model calculates a separate attention distribution for each token position.
02
Weights control contribution
A larger weight means that source contributes more to this weighted mixture.
03
It is learned, not hand-written grammar
The mechanism learns useful patterns from training; these weights are not automatically human explanations.
03 / WHY IT MATTERS
The problem attention helps solve
A sequence contains relationships spread across positions. A single fixed-size summary can become a bottleneck. Attention provides a way for a token to retrieve and combine relevant information from other positions as needed.
Without dynamic attention
Earlier sequence models often had to carry information forward through a recurrent state. Important details could become harder to preserve as sequences grew longer.
With attention
A token can directly compare with other token representations and build a context-aware mixture. In Transformers, this operation can be applied across positions in parallel during training.
04 / STUDY PATH
Explore the subtopics
Start with the intuition, then inspect the mechanism and examples. Each card opens the existing lesson file in this folder.
Interactive end-to-end walkthrough: tokens → embeddings → Q·K·V → scores → softmax → context → Transformer.
INTERACTIVE LAB · 19 STAGES↗
01Ask What Matters Now
The model starts with a token and asks which earlier pieces matter for interpreting it.
ESSENTIAL↗ 02Attention as “Look Back When Needed”A token can use information from other tokens instead of relying only on a fixed summary.
ESSENTIAL↗ 03Attention SimulatorExplore how a focus token distributes attention across the sentence.
BUILD↗ 04Compression BottleneckWhy squeezing a whole sequence into one fixed-size state loses useful detail.
ESSENTIAL↗ 05Core IntuitionThe central idea: compute relevance dynamically for the current token.
ESSENTIAL↗ 06Dynamic FocusThe focus changes depending on which token is being processed.
ESSENTIAL↗ 07Focus: Assign ImportanceTurn relevance scores into weights that control how much information is used.
SUPPORTING↗ 08Predict Better OutputHow context-aware representations can improve the next prediction.
SUPPORTING↗ 09Same Sentence, Different FocusDifferent target words can attend to different parts of the same sentence.
ESSENTIAL↗ 10Translation ExampleSee how attention can connect words across languages during translation.
SUPPORTING↗ 11Understand & Build ContextCombine information from relevant tokens into a context-aware representation.
ESSENTIAL↗ 12Why Attention Was NeededThe limitations that motivated attention as a key mechanism.
ESSENTIAL↗ 13Why Not All Words Matter Equally?Some words are more useful than others for a particular interpretation.
ESSENTIAL↗