Large Language Models — My Understanding
Everything on this page is written by me, as I understand it. Capturing the big picture of how LLMs train, scale, and generate answers during inference.
🎓 The Student Exam Analogy #charan
1. Training Phase (Studying at home): A student practices math problems like 5 + 2 = 7. While studying, he makes errors, corrects his thinking, and internalizes the underlying logic of addition. In an LLM, this is backpropagation updating the billions of weights.
2. Inference Phase (The Exam Hall): On exam day, he gets a brand new question he never saw in his practice book: 9 + 7 = ?. Because his brain mastered the underlying logic, he accurately solves it as 16 without having memorized that exact combination.
During Inference:
- The model's weights are completely frozen (no learning or updates happen).
- The model takes the prompt, converts it to tokens, passes it through Transformer layers, and outputs probability scores for the next token.
- It generates output iteratively, one token at a time (auto-regressive generation).
Modern LLMs (like GPT-4, Llama, Claude) use a Decoder-Only Transformer architecture composed of stacked blocks:
- Multi-Head Self-Attention: Allows each token to gather context from all allowed prior tokens in the sequence.
- Feed-Forward Network (MLP): A dense non-linear layer after attention that processes and transforms token representations (acting like factual memory).
- Residual Connections & LayerNorm: Prevents gradients from vanishing and keeps signals stable across 32 to 96+ layers.
When people say a model has 7 Billion (7B) or 70 Billion (70B) parameters, what are they?
- Parameters are the numerical weights inside the matrices ($W_Q, W_K, W_V, W_O, W_E$, MLP layers).
- Each parameter is typically a 16-bit float (2 bytes). A 7B model takes ~14 GB of VRAM/memory just to load the weights.
- These weights represent everything the model learned during its massive pre-training run.
The Context Length is the maximum number of tokens (e.g. 8k, 32k, 128k, 1M tokens) the model can hold in its attention window at a single moment.
Because self-attention compares all tokens against each other ($O(N^2)$ complexity), longer context windows require techniques like RoPE (Rotary Position Embeddings), FlashAttention, and KV-caching to run efficiently during inference.
According to the Chinchilla Scaling Laws, model performance improves predictably when you scale three factors in balance:
- Number of Parameters ($N$): How large the neural network is.
- Dataset Size ($D$): How many trillions of high-quality tokens it is trained on.
- Training Compute ($C$): FLOPs of GPU/TPU power used.
Things I'm still exploring
- How does KV-caching speed up autoregressive inference by avoiding redundant attention recalculations?
- What are the trade-offs between Quantization (FP16 → INT8 → INT4) and model perplexity?