Self-Attention

For input sequence $X \in \mathbb{R}^{N \times d_{\text{model}}}$, projection matrices $W_Q, W_K, W_V \in \mathbb{R}^{d_{\text{model}} \times d_k}$ generate:

Q = X W_Q, \quad K = X W_K, \quad V = X W_V
\text{Scores} = \frac{Q K^T}{\sqrt{d_k}} \in \mathbb{R}^{N \times N}
\text{Attention Matrix } A = \text{Softmax}(\text{Scores})
\text{Context Output } Z = A V \in \mathbb{R}^{N \times d_k}

The scaling term $\frac{1}{\sqrt{d_k}}$ prevents dot products from growing excessively large in high dimensions, which would otherwise push the softmax function into regions with tiny gradients (gradient vanishing).