Pretraining

How models learn from billions of tokens

🔴 MUST

🧠 Understand👁️ Visualize✏️ Calculate💻 Code⚖️ Compare🗣️ Explain

11 subtopics — click to study

Training Data 🔴 Dataset Preparation 🟡 Tokenization 🔴 Input / Target Creation 🔴 Batching 🟡 Forward Pass 🔴 Loss 🔴 Backpropagation 🔴 Gradient Descent 🔴 Learning Rate 🟡 Checkpoints 🟢

🧪

RASL Pull · Sprint S-05 — Parallel Learning

You're reading this because RASL is blocked on: how do checkpoints work — what does save/restore actually mean in code?

Checkpoints save model state (all weights) so training can be resumed after a crash. RASL's state_tracker uses the exact same save/restore pattern — snapshot execution state after each verified action, restore on failure. Return to mini-llm-lab/train.py and continue building. Read only what unblocks you.

🔨 Lab: M-06 → train.py with checkpoint saves 🔁 Checkpoint pattern = RASL execution state snapshots 📋 View full parallel plan →

← Back to Roadmap