How self-attention creates a context-aware vector

One token. A whole sentence of context.

Let's use a tiny numerical example to see exactly how self-attention takes a token's starting vector and creates a context-aware vector.

We'll use the sentence:

“The cat eats fish.”

Our focus will be the token cat.

The central idea is that cat starts with its own embedding, but self-attention lets it gather information from The, eats, and fish. It then combines that information into a new representation.

Start with the embedding vectors

For simplicity, imagine each token has a vector with just two numbers. These numbers are illustrative, not actual embeddings from a trained model.

Let's call the embedding of cat:

This is the starting representation. It doesn't yet incorporate information gathered from the other tokens in this sentence.

Create queries, keys, and values

Self-attention transforms each input vector into three vectors:

• Query (Q): What information is this token looking for?
• Key (K): What information can this token match against?
• Value (V): What information does this token contribute if selected?

These are calculated using learned weight matrices:

To keep the arithmetic easy, let's assume all three weight matrices are identity matrices:

That means, in this toy example, each query, key, and value equals its original embedding.

For cat:

Compare cat with every token

The query for cat is:

To decide which tokens are relevant, self-attention computes the dot product between this query and each token's key.

For example, the score for eats is:

A larger score means a stronger match between the query and key in this example.

In a real Transformer, these scores reflect learned relationships, not necessarily human-defined semantic relevance.

Turn the scores into attention weights

Before applying softmax, the standard attention formula scales the scores by the square root of the key dimension, √2.

The scaled scores become approximately:

Softmax converts them into positive weights that sum to 1:

How much attention does cat give each token?

Illustrative weights from our chosen vectors and identity matrices.

Notice that cat gives the most weight to itself, followed by eats.

Combine the value vectors

This is the crucial step. Self-attention uses the weights to calculate a weighted sum of the value vectors.

Recall our values:

The new attention output for cat is:

Calculate the first coordinate:

Calculate the second coordinate:

Therefore, approximately:

The numbers changed because the new vector combines information from multiple tokens, according to the attention weights.

Is this the final contextual representation?

Not quite. We've calculated the self-attention output, which is one part of a Transformer layer.

In a typical Transformer block, the model then:

1. Combines the attention output with the original input through a residual connection.
2. Applies normalization.
3. Processes the result through a feed-forward neural network, usually with another residual connection and normalization.

The resulting vector becomes input to the next layer. Repeating these operations can produce increasingly contextual representations.

From a starting vector to a contextualized vector

One subtlety: the starting embedding is already learned. Self-attention doesn't create meaning from nothing; it transforms and contextualizes an existing representation.

A useful next step: explore why Q, K and V are different in real Transformers, or how residual connections preserve the original embedding.