🧱 One-Hot vs Dense Vectors
1. What is a one-hot vector?
Suppose the vocabulary is:
["cat", "dog", "car", "apple"]
Then each word gets a vector with one 1 and the rest 0.
Example:
cat → [1, 0, 0, 0]
dog → [0, 1, 0, 0]
car → [0, 0, 1, 0]
apple → [0, 0, 0, 1]
That is why it is called one-hot.
Only one position is active.
2. Why one-hot is useful
One-hot vectors are useful because they:
- clearly represent identity
- are easy to understand
- show how tokens can become vectors
So they are good for learning the idea of vectorization.
3. Why one-hot is not enough
Problem 1: No similarity
In one-hot space:
cat → [1, 0, 0, 0]
dog → [0, 1, 0, 0]
car → [0, 0, 1, 0]
cat and dog are just as different as cat and car.
That is wrong for meaning.
Problem 2: Huge size
If your vocabulary has:
- 10,000 tokens → vector length 10,000
- 50,000 tokens → vector length 50,000
- 100,000 tokens → vector length 100,000
That becomes very inefficient.
Problem 3: Sparse representation
Most values are zero.
Example:
[0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
This is called a sparse vector.
Sparse vectors waste space and do not naturally capture relationships.
4. Dense vectors are better
Now compare with embeddings:
cat → [0.12, -0.44, 0.91, 0.08]
dog → [0.10, -0.39, 0.88, 0.11]
car → [-0.72, 0.31, 0.05, -0.40]
These are dense vectors because most positions contain useful values.
Benefits:
- smaller size
- more expressive
- can capture similarity
- can be learned from data
5. Intuition: identity vs meaning
A one-hot vector says:
"This token is token number 3."
A dense embedding says:
"This token behaves like these other tokens in useful ways."
That is the big difference.
6. Practical mental model
Think of one-hot vectors like student roll numbers.
Example:
- Student A = roll 1
- Student B = roll 2
- Student C = roll 3
Roll numbers identify students.
But they do not tell you:
- who likes math
- who is similar to whom
- who has related strengths
Dense embeddings are more like learned profiles.