🧭 AI Roadmap · Prerequisites
00 · Essential Math & Intuitive Foundations
You do not need abstract pen-and-paper proofs or a 4-year degree in pure mathematics to master modern AI. You need 4 intuitive, applied math pillars that explain every vector, attention matrix, gradient step, and probability curve in LLMs.
🧙♂️ The Master Mental Model
Brain vs. Open-Book
Think of the LLM as a "Wise Scholar in a Library"
Imagine an elder scholar who spent years reading 100,000 books. When you ask them a question, there are two distinct ways they answer:
🧠 1. The Scholar's Memory (Parametric Weights)
The scholar closes their eyes and speaks from compressed memory formed during years of study.
- Strength: Instant intuition, broad concepts, reasoning across themes.
- Weakness: Fuzzy on exact numbers/dates; might hallucinate false details.
📖 2. The Open-Book Desk (Non-Parametric / RAG)
You hand the scholar an exact open textbook page on their desk (the Context Window).
- Strength: 100% accurate quotes, latest numbers, verified citations.
- Weakness: Limited by desk size (Context Window token budget).
💡 The Core Realization
Every modern Large Language Model is ultimately a mathematical pipeline: Text is converted to Vectors (Linear Algebra), transformed through multi-head Attention (Matrix Multiplications), normalized into Probabilities (Softmax & Statistics), and optimized via Gradients (Calculus). Master these four concepts and the entire AI architecture clicks.
01
Linear Algebra: Vectors, Matrices & Tensors
The Language & Memory of Neural Networks
Computers cannot read characters like 'apple' or 'dog'. Neural networks operate exclusively on continuous numbers arranged in geometric dimensions:
- Scalar (0D): A single number (e.g. learning rate $\eta = 0.001$, loss $\mathcal{L} = 2.45$).
- Vector (1D): An array of numbers representing a concept in coordinate space (e.g. a token embedding with $d_{\text{model}} = 4096$).
- Matrix (2D): A grid of numbers representing linear projections, weight layers ($W_Q, W_K, W_V$), or token sequences ($[N \times D]$).
- Tensor (3D+): Multi-dimensional batches (e.g. $[\text{Batch}, \text{Heads}, \text{SeqLen}, d_k]$).
🎯 Exactly Where This Helps in AI:
01 · Tokens & Embeddings
Translating discrete Token IDs into dense $1 \times d$ concept vectors.
03 · Self-Attention ($QK^T$)
Multiplying Query & Key matrices to produce pairwise token attention grids.
04 · Feed-Forward Expansion
Scaling vectors from $d_{\text{model}} \to 4d_{\text{model}}$ via GEMM weight matrices.
07 · Model Quantization
Compressing 16-bit float matrix weights down to 8-bit or 4-bit integers.
⚡ Interactive Matrix Transformation Explorer
🔍 Behind-the-Scenes: Why GPU Tensor Cores Dominate AI
Modern GPUs (NVIDIA H100, B200) contain dedicated Tensor Cores engineered to do one operation at lightning speed: Fused Multiply-Add (FMA) $\mathbf{D} = \mathbf{A} \times \mathbf{B} + \mathbf{C}$. 99% of training and inference compute is pure matrix multiplication (GEMM).
02
Calculus & Optimization: How Models Learn
Gradients, Loss Functions & BackpropagationWhen an LLM is first initialized, its weights are completely random noise. How does it learn grammar, coding, and reasoning? Through Optimization Calculus:
- Loss Function ($\mathcal{L}$): A mathematical scoring metric (e.g. Cross-Entropy Loss) comparing the model's predicted probability distribution against the true target token.
- The Derivative / Gradient ($\nabla \mathcal{L}$): Tells us the slope: "If I increase weight $w_{i,j}$ by a tiny fraction, will the error go up or down?"
- Backpropagation (Chain Rule): Propagates errors backward through hundreds of Transformer layers via calculus chain rule $\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \cdot \frac{\partial y}{\partial x}$.
- Gradient Descent & AdamW: Updates weights in the opposite direction of the gradient: $W_{\text{new}} = W_{\text{old}} - \eta \cdot \nabla \mathcal{L}$.
🎯 Exactly Where This Helps in AI:
01 · Word2Vec & Embeddings
Adjusting vector dimensions so synonyms attract and antonyms repel.
04 · LLM Pretraining
Updating 70B+ parameters across trillions of tokens to minimize cross-entropy loss.
07 · LoRA & Fine-Tuning
Freezing base weights and optimizing low-rank gradient matrices ($A \times B$).
08 · RLHF & DPO
Steering model preferences via reward gradients and KL-divergence penalties.
⚡ Interactive Gradient Descent Slider
Loss Curve $\mathcal{L}(w) = (w - 3)^2 + 1$
03
Probability & Statistics: How Models Decide
Softmax, Temperature, Sampling & EntropyAn LLM never outputs plain text directly; its final layer produces a vector of raw unnormalized scores called Logits across the entire vocabulary (e.g. 128,000 floats). Probability theory turns those logits into text:
- Softmax Normalization: Exponentiates logits to ensure positive values and divides by the sum: $P(y_i) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$, converting scores into a probability distribution summing to 1.0 (100%).
- Temperature ($T$): Scaling factor controlling distribution flatness. Low $T$ ($\to 0.1$) sharpens probabilities for deterministic factual answers; high $T$ ($\to 1.2$) flattens the curve for creative diversity.
- Top-$k$ & Top-$p$ (Nucleus) Sampling: Pruning the probability tail to avoid incoherent low-probability hallucinations.
🎯 Exactly Where This Helps in AI:
03 · Softmax Attention
Converting raw dot products into percentage attention weights ($a_{ij}$).
04 · Inference & Text Generation
Deciding which word to pick next using temperature, greedy search, or top-$p$.
06 · Prompt Engineering
Configuring temperature & frequency penalties in API payloads (OpenAI, Anthropic).
08 · Mixture of Experts (MoE)
Top-$k$ Softmax gating routers dynamically choosing which 2 of 8 experts activate.
⚡ Interactive Softmax & Temperature Sandbox
Temperature ($T$):
0.7
Prompt context: "The sky is full of dark ____"
04
High-Dimensional Geometry: Similarity & Distance
Vector Space, Cosine Similarity & Metric SearchOnce text is embedded into vectors, semantic meaning becomes spatial geometry. Words or documents with similar meanings point in similar directions in multi-thousand-dimensional hyperspace:
- Dot Product ($\vec{a} \cdot \vec{b} = \sum a_i b_i$): Measures directional alignment and magnitude combined (used inside Self-Attention).
- Cosine Similarity ($\cos \theta = \frac{\vec{a} \cdot \vec{b}}{\|\vec{a}\| \|\vec{b}\|}$): Measures the angular angle between vectors regardless of text length (normalized between $-1.0$ and $+1.0$).
- Euclidean Distance ($L_2 = \sqrt{\sum (a_i - b_i)^2}$): Straight-line geometric distance between vector endpoints.
- The Curse of Dimensionality: In 1536 dimensions, all vectors tend to become orthogonal unless specifically trained into tight semantic manifolds.
🎯 Exactly Where This Helps in AI:
01 · Semantic Vector Space
Concept vector arithmetic ($\vec{v}_{\text{King}} - \vec{v}_{\text{Man}} + \vec{v}_{\text{Woman}} \approx \vec{v}_{\text{Queen}}$).
05 · Vector Databases & RAG
Finding top-$K$ document chunks matching a user query in milliseconds.
05 · Rerankers & Bi-Encoders
Scoring relevance between query embeddings and candidate document vectors.
08 · Multi-Modal Embeddings (CLIP)
Aligning image vectors and text vectors into the exact same shared geometric space.
⚡ Interactive 2D Vector Angle & Similarity Sandbox
$\cos(\theta) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}$