Multi-Head Attention

Parallel attention heads capturing different relationship types

🔴 MUST

🧠 Understand👁️ Visualize✏️ Calculate💻 Code⚖️ Compare🗣️ Explain🔨 Build

⚡ Interactive Visualizer Parallel Feature Subspaces

Multi-Head Attention: Specialization Dissector

Why multiple heads? A single attention head can only compute one relationship pattern per token at a time. Multi-Head Attention splits the $d_{\text{model}}$ vector across $h$ parallel subspaces, allowing each head to specialize in distinct linguistic patterns simultaneously.

Head 1: Syntactic Tracking ($\mathbf{W}_{Q1}, \mathbf{W}_{K1}, \mathbf{W}_{V1}$) Subspace $d_k = d_{\text{model}} / h = 64$

Final Step: Concatenation & Output Projection

$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) \mathbf{W}^O$

[Head 1: 64d] + [Head 2: 64d] + [Head 3: 64d] $\times \mathbf{W}^O \to$ 512d Output

5 subtopics — click to study

Why Multiple Heads? 🔴 Different Relationships 🟡 Parallel Attention 🔴 Concatenation 🔴 Linear Projection 🔴

🧪

RASL Pull · Sprint S-04 — Parallel Learning

You're reading this because RASL is blocked on: why check multiple evidence signals in the verifier — why not just one similarity score?

Multi-head attention shows why looking at the same data through multiple parallel "questions" produces richer representations. RASL's verifier uses both cos_sim and precision@k for the same reason. Return to reliable-agent-lab/agent/verifier.py and continue building. Read only what unblocks you.

🔨 Lab: M-05 → MultiHeadAttention wrapper 🔁 Multi-head intuition → verifier uses cos_sim + precision@k 📋 View full parallel plan →

← Back to Roadmap