// INSIDE TRANSFORMER · SIGNAL FLOW
AI JOURNEY / TRANSFORMER LAB · An interactive, illustrative model of how token representations exchange information and become contextual.
LIVE EXPERIMENT · SELF-ATTENTION

Inside a Transformer: follow the signal

Choose a sentence, select a token, and watch its query compare with every token key. The resulting attention weights mix value vectors into a context-aware representation.

TOY MODEL · TRANSPARENT MATH
ATTENTION LAB · LIVE CONTROLS Choose the sentence, query token, head, and layer before exploring the cards below.

01 / Token-to-token information flow

Follow the glowing streams into the selected query token: each source token contributes information according to its attention weight. Brighter, thicker routes carry more weight.

02 / Query, Key, Value

Small vectors are shown for teaching. Real models use learned projection matrices and much larger vectors.

QUERY · what I seek
KEY · what I offer
VALUE · what I pass on

Context vector (first dimensions)

03 / Attention matrix · selected query row

One row of the attention matrix: the selected query token distributes 100% of its attention across the available tokens. Brighter cells carry more weight. Select a cell to make that token the new query.

QUERY · — LOWHIGH
Cell number = attention weight (%) TOTAL · 100%

04 / Layer-by-layer representation

This simplified view illustrates repeated contextual mixing, not a dump of a trained model's hidden states.

The real Transformer calculation
Scores = QKᵀ / √dₖ → softmax across keys → weighted sum of V. In decoder-only language models, a causal mask prevents a token from reading future tokens. This demo focuses on intuitive attention patterns and labels all generated values as illustrative.
MODEL DEPTH · SCROLL TO EXPLORE
ARCHITECTURE VIEW · 6 REPEATED BLOCKS
BLOCK 06
Transformer layer 6
Multi-head attentionResidual + NormFeed-forward MLP
↑deeper
↑ CONTEXT REFINED
BLOCK 05
Transformer layer 5
Multi-head attentionResidual + NormFeed-forward MLP
05/ 06
↑
BLOCK 04
Transformer layer 4
Multi-head attentionResidual + NormFeed-forward MLP
04/ 06
↑
BLOCK 03
Transformer layer 3
Multi-head attentionResidual + NormFeed-forward MLP
03/ 06
↑
BLOCK 02
Transformer layer 2
Multi-head attentionResidual + NormFeed-forward MLP
02/ 06
↑
BLOCK 01
Transformer layer 1
Multi-head attentionResidual + NormFeed-forward MLP
01/ 06
↑
INPUT
Token + position embeddings
Token IDsVector representations
↓start
↑
OUTPUT
Final norm → vocabulary scores
LM headNext-token probabilities
✓predict
ONE BLOCK, REPEATED
Depth turns
tokens into
context.

Each block receives the representations from the block below. Attention mixes information across tokens; the MLP transforms each token's representation; residual paths help preserve and refine the signal.

SELECTED STAGE · INPUT

Token embeddings

Tokens become vectors and receive positional information before entering the first Transformer block.

Click any block to inspect itStage 0 / 7
01 · MULTI-HEAD ATTENTIONLets each token gather useful information from other tokens. Different heads can learn different relationships.
02 · RESIDUAL + NORMAdds the incoming signal back and stabilizes its scale, helping information and gradients travel through deep stacks.
03 · FEED-FORWARD MLPTransforms each token's updated vector independently, adding nonlinear feature processing.
04 · REPEAT × 6Each block refines the representation. The six blocks here are a teaching illustration, not a claim about every model's depth.
Educational simulation: the sentence patterns, tiny vectors, attention weights, and layer changes are generated locally in this page. They are not extracted from GPT, Ollama, or a trained neural network. The attention math is computed from the displayed toy vectors.