🔄 Static vs Contextual Embeddings

1. Static embeddings

A static embedding gives the same vector every time a token appears.

Example:

bank → always the same vector

This was common in older systems like:

That means:

river bank
bank account

would still start from the same token-level embedding for bank.

This is useful, but limited.

Some words change meaning depending on context.

Examples:

If one token always gets one fixed vector, the model cannot fully represent those differences at the embedding stage.

So static embeddings are good for general similarity, but weak for context-sensitive meaning.

Contextual embeddings change depending on surrounding words.

Example:

bank in "river bank"   → one representation
bank in "bank account" → another representation

This is much closer to how language actually works.

Modern transformer-based models produce contextual representations by letting tokens interact with surrounding tokens.

That is where vectorization starts connecting directly to attention.

Contextual embeddings help models:

This is one reason transformers became so powerful.

At the very input layer, transformer models still begin with token embeddings from an embedding table.

But after attention layers process the sequence, the token representations become contextualized.

So a useful mental model is:

input embedding → initial token vector
attention + layers → contextual representation

That means contextual embeddings are not just “lookup table vectors.” They are refined representations shaped by the full sequence.

This topic is the bridge between:

Without this distinction, it is easy to think embeddings are always fixed.

But in modern AI, the most useful token representations are often contextual.