Static vs Contextual Embeddings

Every token in a language model starts its life as a static vector — a row pulled from an embedding table by token ID. But this single fixed vector cannot capture the fact that language is deeply contextual. Two kinds of embeddings exist, and understanding the difference is essential to understanding why transformers work the way they do.

Natural language is full of polysemous words — single words with multiple distinct meanings. A static embedding can only encode one vector per token, so it inevitably averages over all the word's meanings. This averaging loses critical information.

The clearest analogy: a static embedding is like a passport photo. No matter what you are doing today — whether you are at a beach, in a courtroom, or running a marathon — your passport always shows the same picture. The photo does not change based on context.

A contextual embedding is like a live camera feed. Your appearance (representation) shifts based on what surrounds you — who else is in the frame, what activity is happening, what environment you're in. The system sees you in context.

Static embedding methods train by predicting word co-occurrences over a large corpus. The result is a fixed lookup table: one vector per vocabulary item, trained once, used at inference without modification.

Strengths of static embeddings: fast to compute at inference time (pure table lookup), small memory footprint, good baseline for general word similarity tasks, easy to inspect and visualise.

Modern transformer models (BERT, GPT, T5, LLaMA) produce contextual representations. The initial token embedding is still static — it comes from the same kind of embedding table — but attention layers transform it based on surrounding tokens. The final hidden state for a token is contextual.

This is precisely why the Attention mechanism exists — it is the engine that converts static embeddings into contextual representations by allowing each token to look at its neighbours and update accordingly.

The full pipeline from raw text to contextual hidden state, step by step:

1. Tokenize: Split raw text into tokens (WordPiece, BPE, SentencePiece…).
2. Token ID lookup: Convert tokens to integer IDs using the vocabulary.
3. Embedding table lookup: Each ID indexes one row in the embedding matrix $E \in \mathbb{R}^{V \times d}$. This is the static embedding.
4. Positional encoding: Add position information — transformer has no inherent notion of order.
5. Attention layers: Each layer computes attention scores between all token pairs, allowing each token's representation to be updated based on context.
6. Output: Final hidden states — one $d$-dimensional vector per token, fully contextual. These are what downstream tasks use.

Consider the word "bank" in two sentences with completely different meanings. We trace what happens at each layer.

The Mermaid diagram below shows how both sentences start from the identical static embedding for "bank" but end up with different contextual representations after the attention layers operate.

The dashed lines show the key contrast: the static embeddings are identical (same starting point), while the final hidden states are different (context has differentiated them). This is the mechanism attention provides — and the reason contextual representations require it.

The following Python code demonstrates that BERT gives "bank" genuinely different hidden states in two sentences. We extract the hidden state for the "bank" token from each sentence and compute cosine similarity — it should be well below 1.0, showing the representations have diverged.

import torch
from transformers import BertTokenizer, BertModel
import torch.nn.functional as F

# ── Load model (downloads once, ~440 MB) ────────────────────────────────────
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")
model.eval()

def get_token_hidden_state(sentence: str, target_word: str) -> torch.Tensor:
    """
    Extract the final-layer hidden state for the first occurrence
    of target_word in sentence.

    Returns a 1D tensor of shape (768,).
    """
    inputs = tokenizer(sentence, return_tensors="pt")
    token_ids = inputs["input_ids"][0].tolist()
    tokens = tokenizer.convert_ids_to_tokens(token_ids)

    # Find the index of the target word token
    target_idx = None
    for i, tok in enumerate(tokens):
        if tok == target_word.lower():
            target_idx = i
            break

    if target_idx is None:
        raise ValueError(f"'{target_word}' not found as a single token in: {tokens}")

    with torch.no_grad():
        outputs = model(**inputs)

    # outputs.last_hidden_state shape: (1, seq_len, 768)
    hidden_state = outputs.last_hidden_state[0, target_idx, :]  # (768,)
    return hidden_state

# ── Two sentences with "bank" ────────────────────────────────────────────────
s1 = "The bank called about my account."
s2 = "The river bank flooded last night."

h_bank_fin = get_token_hidden_state(s1, "bank")   # financial context
h_bank_geo = get_token_hidden_state(s2, "bank")   # geographic context

# ── Cosine similarity ────────────────────────────────────────────────────────
cos_sim = F.cosine_similarity(h_bank_fin.unsqueeze(0),
                               h_bank_geo.unsqueeze(0)).item()

print(f"'bank' in financial context: {h_bank_fin[:5].tolist()}...")   # first 5 dims
print(f"'bank' in geographic context: {h_bank_geo[:5].tolist()}...")

print(f"\nCosine similarity between the two 'bank' vectors: {cos_sim:.4f}")
# Expected: something like 0.72 - 0.88 (well below 1.0, showing divergence)

# ── For comparison: word2vec would give cosine = 1.0 ────────────────────────
# Word2Vec: one fixed vector. Same vector = cosine similarity = 1.0.
# Static embeddings cannot distinguish these two contexts at all.

Contextual embeddings are the engine behind most modern NLP systems. Here are four areas where the context-dependent nature of the representation is not just helpful — it is required.

Here is the key insight that ties the entire vectors roadmap to the attention roadmap. The reason contextual embeddings require attention is not arbitrary design — attention is precisely the mechanism that allows a token's representation to be influenced by its neighbours.

Without attention:

• Each token's representation would remain fixed at its static embedding after lookup.
• "bank" after "river" would look identical to "bank" after "account".
• There would be no mechanism for polysemy resolution, coreference, or any sentence-level understanding.

With attention:

• Each token computes a weighted sum over all other tokens' representations, weighted by relevance (query–key dot products).
• A token like "bank" after "river" will put high attention weight on "river", pulling its representation toward the geographic sense.
• After multiple layers, the hidden state has incorporated rich contextual information from the entire sequence.

This is the complete conceptual bridge. The entire Attention branch of the AI Journey roadmap is about how this transformation happens — how query/key/value matrices are formed, how attention scores are computed and normalised, how multi-head attention captures multiple relationship types simultaneously, and how all of this stacks into a transformer.

Common misconceptions to tackle:

• "BERT has no static embeddings — it's all contextual." Wrong. BERT absolutely has a static embedding table. Every token still starts as a fixed vector from that table. Attention then builds on top of it.
• "Contextual is always better than static." Depends on the task. For fast retrieval over millions of documents, static embeddings require no forward pass. For meaning-sensitive tasks, contextual wins.
• "I can compare contextual embeddings from different models directly." No. Contextual hidden states are not interoperable across models — different architectures, different training, different representation spaces.
• "The initial embedding table is useless once attention runs." No. The embedding table is the foundational input. Its quality shapes the entire representation that attention builds on. Bad initial embeddings make the transformer's job harder.
• "Word2Vec is obsolete and wrong." It's limited for contextual tasks, but not wrong — it's a fast, compact representation that's still useful for many similarity and retrieval tasks where a full transformer forward pass is too expensive.

Summary