Attention — My Understanding

Step-by-Step Walkthrough: The "Plant" Example #charan

Consider the sentence: "I plant an idea into Rahul's mind."

• Step 1 — Initial Embedding: Tokenizer assigns token ID plant → 48365. It gets an initial lookup vector (meaning + position). At this stage, it holds a generic dictionary representation that does not yet reflect its role in this sentence.
• Step 2 — Self-Attention: The model calculates how much attention plant should pay to surrounding tokens. It attends heavily to idea, into, and mind, which establish its meaning as an action/metaphor rather than a physical organism.
• Step 3 — Contextual Representation: The attention mechanism blends relevant information together to create an updated representation for plant.

Compare this with "The green plant needs water": both start with the exact same token ID and initial vector, but attention gives each sentence a completely different contextual representation.

Here is an illustrative example of how the numerical coordinates shift after attention mixes context:

Crucial Correction: Do the vectors move closer together? #charan

• ❌ Not geometric pulling: Attention does not simply pull related word vectors closer to each other in geometric space.
• ✅ Information gathering: Attention learns how much information one token should take from other tokens. The resulting vector changes in both direction and magnitude.

Two Technical Nuances to Keep in Mind:

1. How values are computed: Attention doesn't just edit numbers directly. It projects vectors into Queries (Q), Keys (K), and Values (V), calculates attention weight scores (via Softmax), and computes a weighted sum of the Values.
2. Not a human label: The contextual vector is not an explicit tag saying "verb" or a dictionary definition. It is a continuous numerical coordinate in multi-dimensional space that encodes grammatical and semantic patterns.

When the model processes a sentence, it doesn't process tokens in isolation. For each token position, it asks: "Which other words in this sentence matter right now to understand my role?"

For example, in "Adavitha is playing with Charan in the park":

• When processing playing, the most important words are Adavitha (who is doing it) and Charan (who she is playing with).
• When processing park, the most important connection is in and playing (where the action takes place).

Before Transformers and Attention, older models like RNNs tried to squeeze an entire sentence word-by-word into a single fixed-size hidden vector (the bottleneck).

By the time an RNN reached the 20th word, details from the 1st word were often faded or completely lost.

Attention eliminates the bottleneck: Instead of relying on a compressed summary, any token can directly reach back and connect with any other token in the sentence with zero distance decay.

Attention is dynamic because the focus changes depending on which word is currently asking the question (the Query):

• Subject asking: Adavitha looks at playing (What is she doing?).
• Action asking: playing looks at Adavitha and Charan (Who are the participants?).
• Location asking: park looks at in and playing (What happens here?).

Every token gets its own unique attention distribution across the entire sentence.

In a sentence like "The large, ancient green plant in the corner of the garden needs water", when predicting what happens to plant, words like needs and water carry high attention weights.

Raw comparison scores can be negative or zero, so the model passes them through the Softmax function (scaled by $\sqrt{d_k}$) to convert them into positive probabilities that add up to 1.0 (100%).

See the full formula & temperature breakdown in Self-Attention → Softmax Weights & Scaling ↗.

The distinction: Breadth vs. Depth #charan

There are two separate dimensions working together:

1. Multi-Head Attention (Breadth / Parallel within ONE layer)

In a single layer, multiple attention heads inspect the sentence simultaneously for different types of relationships:

• Head 1 (Grammar/Subject): Connects plant ↔ I (Who is performing the action?).
• Head 2 (Object/Direct Target): Connects plant ↔ idea (What is being planted?).
• Head 3 (Location/Recipient): Connects plant ↔ Rahul's mind (Where is it being planted?).

Each head has its own set of Query, Key, and Value matrices. At the end of the layer, their insights are concatenated and merged.

2. Multi-Layer Attention (Depth / Sequential Refinement)

The vector passes through a stack of 12 to 32+ layers sequentially. The numbers in the vector get progressively refined:

• Early Layers (1–4): Recognize local syntax, parts of speech (e.g. plant is acting as a verb here).
• Middle Layers (5–12): Connect clauses, direct objects, and pronoun references.
• Deep Layers (13+): Resolve complex metaphors (e.g. "planting an idea" is about cognitive influence/inception, not biology or soil).

Here is how the entire data flow connects from text to final prediction:

1. Tokens: Raw text is split into subword token IDs (e.g. plant → 48365).
2. Lookup Embeddings: Token ID looks up a fixed numerical row in the embedding table + positional encoding is added.
3. Multi-Head Self-Attention: Parallel heads calculate Q, K, V attention weights for syntax, semantics, and relationships.
4. Multi-Layer Stacking: Stacks of Transformer layers progressively refine token vectors from basic dictionary definitions into deep contextual representations.
5. Final Context Vector: The output holds the exact contextual role of every word in the sentence.
6. Next Token Prediction: Logits are calculated from the final context vector to predict the next word accurately.

• How exactly does Multi-Head Attention split the dimensions across heads (e.g., 768 dimensions split into 12 heads of 64 dimensions each)?
• How do Feed-Forward Networks (FFN) right after Attention store factual knowledge vs contextual relations?