Self-attention: every word can look at every word

Use a real sentence. Each word is a token-position, and each curved line shows one possible attention connection between a query word and a key word.

01 · Select a word and trace its connections

Choose any word. Watch its connections to every word light up.

10 words · 100 connections

Selected query tokencurious

Connections from this token10 possible comparisons

Selected query tokenOther key tokensPossible attention connections

02 · Attention score matrix

Rows are query tokens; columns are key tokens. Word labels appear on both axes.

How to read it: Pick a word from the left-hand side to choose a query row. Move across that row to see how that word compares with every key token along the top.

Example: If “curious” is the query, its row contains a possible score for “curious” against “The,” “cat,” “watches,” and every other word. Those scores help the model determine which context is useful when representing “curious.”

Why n²? With n tokens, there are n rows × n columns = n² cells. Ten tokens therefore create 100 possible token-pair scores.

The highlighted cells show connections for learning purposes; they are not the real attention weights from a trained model.

10 × 10

KEY TOKEN →

Tokens (n)10

Possible pairs (n²)100

Pairs for selected query10

10×10=100

Why quadratic? Each of the n query positions can compare with all n key positions. That creates n × n = n² score positions. With 10 tokens, that's 100 possible pairwise scores.

The connections show possible comparisons, not learned attention weights from a real model.