Vectors — My Understanding
Everything on this page is written by me, as I understand it. Not copied from the lessons — just what I took away after understanding how words and tokens become geometrical arrows in multi-dimensional space.
Computers and neural networks cannot perform linear algebra or matrix multiplications on raw strings or characters. Every word or token must be mapped into continuous real numbers (a vector) so mathematical operations (addition, dot products, transformations) can measure semantic relationships.
Why not use simple One-Hot Encoding where every word gets a vector with a single 1 and 50,000 0s?
- One-Hot Vectors (Sparse): Vector size is massive ($|V| \approx 50,000$). Every word is orthogonal ($\mathbf{u} \cdot \mathbf{v} = 0$).
"king"and"queen"have zero similarity. - Dense Vectors (Embeddings): Vector size is compact ($d = 768$ to $4096$). Every coordinate contains a continuous floating-point value. Words with similar meanings cluster together in geometric space!
In vector space, direction corresponds to meaning:
- Cosine Similarity $= +1.0$ ($0^\circ$ angle): Identical direction / highly synonymous words (e.g.
"thought"and"idea"). - Cosine Similarity $= 0.0$ ($90^\circ$ angle): Orthogonal vectors / completely unrelated topics (e.g.
"plant"and"quadratic"). - Cosine Similarity $= -1.0$ ($180^\circ$ angle): Diametrically opposed directions.
$$\text{Cosine Similarity}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|}$$
The same word can mean different things depending on the sentence:
- "Deposit money in the bank" → bank is a financial institution.
- "Sit on the river bank" → bank is the land beside a river.
A static embedding gives "bank" one fixed vector for both sentences. In the Transformer Attention stage, self-attention rotates and shifts the vector based on surrounding words, producing an updated Context Vector.