📊 Vectorization (Embeddings)

🌟 Start With a Real-World Scenario

Imagine this: You're searching Netflix for "movies like Inception"

Netflix does not just match the exact word Inception. It tries to understand related meaning:

• 🎬 Inception is similar to Interstellar
• 🧠 mind-bending is related to psychological thriller
• 🎭 one creator or style can connect to another similar creator or style

But how? AI cannot work directly with raw words.

The key step is vectorization: converting tokens into mathematical vectors.

"King"    = [0.2, 0.8, 0.1, 0.5, ...]
"Queen"   = [0.3, 0.7, 0.15, 0.45, ...]
"Man"     = [0.5, 0.2, 0.1, 0.3, ...]
"Woman"   = [0.6, 0.1, 0.15, 0.25, ...]

These are not random numbers. They are learned representations.

That is why older embedding demos became famous for examples like:

King - Man + Woman ≈ Queen

Treat that as a useful intuition, not a universal law. The deeper idea is:

similar meanings often get similar vector patterns

This is why vectorization helps with:

• semantic search
• recommendations
• translation
• clustering related content
• later attention and transformer processing

Real examples:

Google Search

• You search: best phone under $500
• The system can still connect that to ideas like budget smartphone
• It is not limited to exact keyword matching

Spotify Recommendations

• You like one song
• The system can find nearby songs in representation space
• Not because the titles match, but because the learned patterns are related

Modern language models

• You say: I'm feeling blue
• The system should not treat blue only as a color
• It needs representations that can later be refined using context

By the end of this folder, you should understand how AI turns tokens into vectors that are useful for meaning.

🗺️ Concept Mind Map

                    VECTORIZATION (EMBEDDINGS)
                              |
        +---------------------+----------------------+
        |                     |                      |
    TOKEN IDS            EMBEDDINGS            CONTEXTUAL USE
        |                     |                      |
    - labels only        - dense vectors       - meaning shifts
    - no similarity      - learned patterns    - depends on context
    - no semantics       - useful geometry     - important for transformers
        |                     |                      |
        +----------- why IDs fail -----------+------+
                                             |
                                   bridge to attention

Key insight:

• token IDs are identifiers
• vectors are usable numeric representations
• similar meanings can become nearby in vector space
• later attention layers build on those vectors

🎯 Why this topic exists

Tokenization teaches the model how to split text into pieces.

Vectorization teaches the model how to represent those pieces numerically in a useful way.

Keep this pipeline in mind:

Text
→ Tokenization
→ Token IDs
→ Embeddings / vectors
→ Attention
→ Transformer layers
→ Predictions

If tokenization teaches AI to read pieces, vectorization teaches AI to represent those pieces in a learnable form.

🧠 Core Concept

The Problem: Tokens Are Just IDs

After tokenization, we may have something like:

"cat" → 42
"dog" → 87
"kitten" → 156

Problems:

1. 42 and 156 are just numbers, not meaning
2. we cannot tell whether cat is closer to dog or car
3. raw IDs do not support useful similarity comparisons

The Solution: Dense Vector Representations

Instead of using only IDs, we map each token to a dense vector:

"cat"    → [0.2, -0.5, 0.8, 0.1, ...]
"kitten" → [0.3, -0.4, 0.7, 0.2, ...]
"dog"    → [0.1, -0.6, 0.9, 0.0, ...]
"car"    → [-0.8, 0.3, -0.2, 0.9, ...]

Now the model has something it can compute with.

Key idea: words used in similar ways often end up with related vectors.

The Distributional Hypothesis

A classic idea behind embeddings is:

"You shall know a word by the company it keeps."

Example:

• The cat sat on the mat
• The dog sat on the mat
• The kitten sat on the mat

Because cat, dog, and kitten appear in similar contexts, training may push them toward related representations.

📍 Simple Analogy: GPS Coordinates

Think of vectors like coordinates on a map.

• Word = city name
• Vector = coordinates
• Similar words = nearby locations
• Distance = rough semantic similarity

This analogy is not perfect, but it helps.

A vector gives the model a position in a learned space where relationships can be compared.

If two tokens are near each other, they may behave similarly in the learned representation.

For the deeper similarity idea, continue with 06-Similarity-in-Vector-Space.md.

🧩 What you will learn in this folder

Use this folder in this order:

1. 01-Why-Token-IDs-Are-Not-Enough.md
2. 02-One-Hot-vs-Dense-Vectors.md
3. 03-Embeddings.md
4. 04-Embedding-Dimensions.md
5. 05-How-Embeddings-Are-Learned.md
6. 06-Similarity-in-Vector-Space.md
7. 07-Static-vs-Contextual-Embeddings.md
8. 08-Vectorization-to-Attention-Bridge.md

This order keeps the topic practical:

• first understand why IDs fail
• then see how vectors solve that
• then learn how embeddings are learned
• then understand similarity and context
• then move toward attention

By the end, you should clearly understand:

• why token IDs are not enough
• what a vector representation is
• why embeddings are learned instead of manually assigned
• why dimensions like 128, 768, or 4096 appear
• how similarity works in vector space
• why contextual embeddings are more powerful than static ones
• how embeddings become the input to attention and transformers

🎉 Practical notes worth remembering

Useful intuition

• Similar meanings often end up near each other in vector space
• Embeddings are learned from usage patterns, not manually written by humans
• Modern models start with embeddings, then refine them using context

Important caution

Embeddings are powerful, but they are not magic truth machines.

A close vector does not always mean:

• exact synonym
• same meaning in every sentence
• perfect human-level understanding

They capture useful statistical structure.

Where the deeper topics live

This overview intentionally does not go deep into everything.

Use these notes for the next layers of understanding:

• similarity and analogy intuition → 06-Similarity-in-Vector-Space.md
• bank-style context changes and static vs contextual behavior → 07-Static-vs-Contextual-Embeddings.md
• how embeddings become inputs to attention → 08-Vectorization-to-Attention-Bridge.md
• positional information in transformers → 3.d-Positional-Encoding.md

🎯 Suggested study style

Do not treat this as a heavy math chapter first.

Study it in this order:

• intuition
• examples
• practical meaning
• only then deeper mechanics

That will keep the topic clear without becoming too theoretical too early.