← 00 · Math Foundations Hub

Linear Algebra for AI

How continuous geometric coordinate arrays (Vectors, Matrices, Tensors) represent every piece of text, memory, and computation inside modern Large Language Models.

🟣 CORE PILLAR 01

🧠 Vectors (1D) 📐 Matrices (2D) 🧊 Tensors (3D+) ⚡ GEMM Multiplication

💡 The Linear Algebra Principle

An AI model never sees words or letters. It sees a vector of continuous floats (e.g., $[0.24, -0.81, 0.55, \dots]$). Every layer of a Transformer is a matrix multiplication that rotates, stretches, and projects these vectors into new conceptual feature spaces.

Scalars, Vectors, Matrices & Tensors

The Hierarchy of AI Data Structures

Every piece of data moving through a neural network has a specific mathematical shape:

Scalar ($0\text{D}$): A single number (e.g., loss $\mathcal{L} = 1.42$, temperature $T = 0.7$).
Vector ($1\text{D}$): A 1D array of numbers with dimension $d_{\text{model}}$ (e.g., GPT-3 embedding has $d = 12,288$ numbers).
Matrix ($2\text{D}$): A 2D grid $[M \times N]$ used for weight projections ($W_Q, W_K, W_V$) or a sequence of tokens $[S \times D]$.
Tensor ($3\text{D}+$): Multi-dimensional container representing batches: $[\text{Batch Size}, \text{Num Heads}, \text{Sequence Length}, d_k]$.

🎯 Where This Directly Applies:

Module 01: Embeddings Converts integer token IDs into $[1 \times 4096]$ continuous vectors.

Module 03: Attention Multiplies Sequence Matrix $[N \times D]$ by Weight Matrix $[D \times d_k]$.

Matrix Multiplication (GEMM) & Feature Projection

How Neural Networks Transform Meaning

General Matrix Multiplication (GEMM) is the fundamental workhorse of all AI hardware. When an input vector $\mathbf{x}$ is multiplied by a weight matrix $\mathbf{W}$, each row of $\mathbf{W}$ acts as a custom feature detector:

$$\mathbf{y} = \mathbf{W} \mathbf{x} + \mathbf{b}$$

In Self-Attention, Query ($\mathbf{Q}$), Key ($\mathbf{K}$), and Value ($\mathbf{V}$) matrices are produced by 3 parallel matrix multiplications against the input token sequence:

$$\mathbf{Q} = \mathbf{X} \mathbf{W}_Q, \quad \mathbf{K} = \mathbf{X} \mathbf{W}_K, \quad \mathbf{V} = \mathbf{X} \mathbf{W}_V$$

⚡ Live $2 \times 2$ Matrix Projection Calculator

Vector Input $x_1$: 1.0 Vector Input $x_2$: 2.0

Vector Dot Product & Matrix Transposition ($Q K^T$)

The Engine of Attention Scores

The dot product between two vectors $\mathbf{u} \cdot \mathbf{v} = \sum_{i=1}^d u_i v_i$ computes directional agreement. When two token vectors point in the same direction, their dot product is large and positive.

In Transformer Attention, transposing Key matrix $\mathbf{K}$ to $\mathbf{K}^T$ allows computing dot products between every single pair of tokens simultaneously in a single GPU operation:

$$\text{Scores} = \mathbf{Q} \mathbf{K}^T \in \mathbb{R}^{N \times N}$$