← 00 · Math Foundations Hub

Probability & Statistics for AI

How language models turn raw logits into probability distributions, decide next words through temperature sampling, and measure uncertainty via entropy and perplexity.

🟡 CORE PILLAR 03

📊 Logits & Softmax 🌡️ Temperature ($T$) 🎯 Top-$k$ & Top-$p$ 📐 Perplexity

💡 The Probabilistic Nature of LLMs

An LLM is not a database of rigid facts; it is a statistical conditional probability calculator: $P(w_t \mid w_1, w_2, \dots, w_{t-1})$. Every sentence it produces is sampled step-by-step from probability distributions shaped by Softmax.

Softmax: From Raw Logits to Probabilities

Ensuring $\sum P(y_i) = 1.0$

The final linear layer outputs raw unconstrained numbers called Logits $z \in (-\infty, +\infty)$. Softmax exponentiates every logit to guarantee positivity and divides by the sum of all exponentiated values:

$$P(y_i) = \frac{e^{z_i / T}}{\sum_{j=1}^V e^{z_j / T}}$$

In Self-Attention, Softmax is applied across row attention dot products to determine how much relative weight each token assigns to other tokens.

Temperature, Top-$k$ & Top-$p$ (Nucleus) Sampling

Controlling Randomness, Hallucination & Creativity

Dividing logits by temperature $T$ reshapes the probability distribution:

Low Temperature ($T \to 0.1$): The highest logit dominates ($P \to 99\%$), resulting in deterministic, greedy, and repetitive answers (great for coding/math).
High Temperature ($T \to 1.5$): Flattens the curve, giving low-probability words a chance to be chosen (great for creative writing, but risks incoherence).
Top-$p$ (Nucleus Sampling): Truncates the pool to only the smallest set of tokens whose cumulative probability exceeds threshold $p$ (e.g. $p=0.90$).

⚡ Interactive Sampling Sandbox

Temperature ($T$): 0.7