Contextual Embeddings

A contextual embedding (also called a contextual representation or hidden state) is a token's vector after the transformer has mixed in information from the surrounding tokens. A static token embedding is the context-free row read straight from the embedding matrix.

The lookup in Token Embeddings depends on one thing only: the token ID. Same ID, same row — always.

• Words are ambiguous: bank, bat, light, play.
• Meaning also changes through grammar and reference: "it" refers to different things in different sentences.
• Language models must predict the next token or answer questions about this sentence, not about the word in general.

Think of the embedding-matrix row as a lump of clay taken from the shelf. Every sentence that contains bank starts with an identical lump.

• In "The bank approved my loan", the neighbours approved and loan press the clay toward a financial shape.
• In "We sat on the bank of the river", sat and river press it toward a riverside shape.
• The shaping rules — how strongly to listen to which neighbour — are the transformer's learned weights. They were fixed during training.

The full side-by-side walkthrough of this example lives on Static vs Contextual Embeddings. This page focuses on where it sits in the pipeline and shows a small calculation you can check by hand.

Write $\mathbf{h}_i^{(0)}$ for the vector a token has when it enters the first transformer layer, and $\mathbf{h}_i^{(\ell)}$ for its vector after layer $\ell$.

Where the knowledge lives

• The learned parameters — the embedding matrix and the transformer weights — were established during training.
• At inference, the same fixed weights process every sentence. What changes from sentence to sentence is the input, and therefore the contextual vectors.
• So the model does not "gain" knowledge at the embedding stage; it applies what it already has to your tokens.

Imagine dimension 0 means "finance-ish" and dimension 1 means "nature-ish". These numbers are invented for teaching; real embeddings are not interpretable axis by axis.

In one simplified attention step, token $i$ builds a new vector from all tokens $j$:

• $\alpha_{ij}$ is how much token $i$ listens to token $j$; the weights over $j$ sum to 1.
• $\mathbf{q}, \mathbf{k}, \mathbf{v}$ are produced from the token vectors by learned weight matrices (see the Q/K/V roadmap topic).
• Because $\alpha_{ij}$ depends on which tokens are present, $\mathbf{h}_i'$ depends on the sentence, even when the starting vector does not.

In the toy above we set $\mathbf{q}=\mathbf{k}=\mathbf{v}=$ the token vector and $d = 2$.

Part A — the toy from the Calculate phase:

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

def softmax(xs):
    m = max(xs)
    es = [math.exp(x - m) for x in xs]
    s = sum(es)
    return [e / s for e in es]

def self_attend(vectors, i):
    """Simplified attention: q = k = v = the vector itself."""
    d = len(vectors[0])
    q = vectors[i]
    scores = [sum(a * b for a, b in zip(q, k)) / math.sqrt(d) for k in vectors]
    w = softmax(scores)
    h = [sum(w[j] * vectors[j][c] for j in range(len(vectors))) for c in range(d)]
    return w, h

bank, loan, river = [0.5, 0.5], [0.9, 0.1], [0.1, 0.9]

for name, ctx in (("loan", loan), ("river", river)):
    w, h = self_attend([bank, ctx], 0)
    print(name, [round(x, 2) for x in w], [round(x, 2) for x in h],
          "finance %.3f nature %.3f" % (cosine(h, [1, 0]), cosine(h, [0, 1])))

# loan  [0.5, 0.5] [0.7, 0.3] finance 0.919 nature 0.394
# river [0.5, 0.5] [0.3, 0.7] finance 0.394 nature 0.919

h_a = self_attend([bank, loan], 0)[1]
h_b = self_attend([bank, river], 0)[1]
print(round(cosine(h_a, h_b), 3))      # 0.724

Part B — a real model (BERT via HuggingFace):

import torch
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased").eval()

def bank_vectors(sentence):
    enc = tok(sentence, return_tensors="pt")
    idx = tok.convert_ids_to_tokens(enc["input_ids"][0]).index("bank")
    with torch.no_grad():
        raw = model.embeddings.word_embeddings(enc["input_ids"])[0, idx]   # embedding-matrix row
        ctx = model(**enc).last_hidden_state[0, idx]                       # contextual vector
    return enc["input_ids"][0, idx].item(), raw, ctx

id_a, raw_a, ctx_a = bank_vectors("The bank approved my loan.")
id_b, raw_b, ctx_b = bank_vectors("We sat on the bank of the river.")

print(id_a == id_b)                              # True  -> same token ID
print(torch.equal(raw_a, raw_b))                 # True  -> same embedding-matrix row
cos = torch.nn.functional.cosine_similarity
print(cos(ctx_a, ctx_b, dim=0).item())           # below 1.0 -> contextual vectors differ

• Classification and tagging: encoder models such as BERT feed contextual vectors to small task heads (sentiment, named entities).
• Next-token prediction: in decoder models such as GPT, the final-layer vector at the last position is projected to scores over the vocabulary.
• Text embeddings for search: sentence-embedding models pool contextual vectors into one vector per text for similarity search (see the Embeddings for Retrieval topic).
• Disambiguation and coreference: resolving which sense or which entity a word refers to is a contextual task.

🚫 Common Misconceptions to Correct

• Misconception: different meaning means a different token ID.
  Reality: One ID can carry many senses; context separates them after the lookup.
• Misconception: the embedding matrix stores every sense of every word.
  Reality: It stores one vector per vocabulary entry. Sense-specific vectors are computed by the transformer.
• Misconception: all knowledge comes from the embedding table.
  Reality: The transformer weights hold a large part of what the model has learned.

📌 Summary — Everything in Six Bullets

• Same ID, same starting vector. The lookup ignores context.
• Context enters in the transformer. Attention mixes information from neighbouring tokens.
• Contextual vectors differ per sentence. That is how bank can mean different things.
• Weights are fixed after training. The vectors are computed fresh for each input.
• Knowledge lives in the weights. Both the embedding matrix and transformer layers contribute.
• Pipeline: Text → Token IDs → Embedding vectors → Transformer computations → Contextual representations.