Calculus & Optimization for AI
How neural networks learn: Loss functions, partial derivatives, the calculus chain rule (Backpropagation), and gradient descent optimization (AdamW).
Loss Functions: Measuring Prediction Error
Cross-Entropy Loss in Language ModelsIn language modeling, the training objective is Next-Token Prediction. When the target token is $y$ and the model predicts probability $P(y)$, the error is calculated using Cross-Entropy Loss:
$$\mathcal{L} = -\log P(y_{\text{target}})$$If the model predicts $P(\text{target}) = 0.99$, $\mathcal{L} = -\log(0.99) \approx 0.01$ (nearly zero penalty). If it predicts $P(\text{target}) = 0.01$, $\mathcal{L} = -\log(0.01) \approx 4.60$ (massive penalty).
Gradients & Backpropagation (The Chain Rule)
How Error Flows Backward Through 96 Transformer LayersBecause Transformer networks are deep composite functions $y = f(g(h(x)))$, finding the gradient for early layers requires the Calculus Chain Rule:
$$\frac{\partial \mathcal{L}}{\partial w} = \frac{\partial \mathcal{L}}{\partial y} \cdot \frac{\partial y}{\partial h} \cdot \frac{\partial h}{\partial w}$$PyTorch automates this through its Autograd computational graph engine, generating backward gradient tensors for all active parameters.
Gradient Descent & Learning Rate Dynamics
Nudging Parameters Down the Loss ValleyOnce the gradient $\nabla \mathcal{L}$ is computed, parameters are updated via:
$$W_{t+1} = W_t - \eta \cdot \nabla \mathcal{L}$$Where $\eta$ is the Learning Rate. If $\eta$ is too high, the model explodes into `NaN`; if too low, training takes decades.