🔁 Self-Attention

Self-attention means:

each token looks at other tokens in the same sequence to build a better representation of itself.

Why it matters

In a sentence like:

"The report reached the manager after several revisions, and it changed the release plan."

When the model processes "it", it should look more strongly at:

• "report"
• the surrounding action and sentence structure

That is self-attention: the token "it" attends to other tokens in the same sentence to build the right meaning.

Core intuition

Every token asks:

• What other tokens are relevant to me right now?
• Which ones should influence my meaning?
• How much weight should each one get?

So each token becomes context-aware.

Without self-attention:

• a token representation is more isolated

With self-attention:

• a token representation includes useful signals from related tokens

Example

Sentence:

"The bank can refuse to lend money."

When processing "bank", tokens like:

• "lend"
• "money"
• "refuse"

help the model understand that "bank" means a financial institution.

Why this was powerful

Older models mainly passed information forward step by step.

Self-attention allowed:

• direct access across the sequence
• better long-range dependency handling
• more parallel computation
• richer context for every token

Live intuition demos

If you want to see the intuition before the real math, run:

• 04-Resources/Code-Examples/03-Attention/1.basic_attention_demo.py
• 04-Resources/Code-Examples/03-Attention/2.attention_brain_simulator.py

These demos do not implement real self-attention with QKV yet.

Instead, they simulate the idea that:

• the current token asks what matters now
• other tokens get different importance
• stronger clues influence the final interpretation more

They are useful as a learning bridge from intuition → real self-attention.

Bridge to next topic

Self-attention is implemented using:

• Query
• Key
• Value

Next:

• 3.b-QKV