🎮 Reinforcement Learning

🌟 Start With a Real-World Scenario

Imagine training a dog:

Bad Training (No Feedback):

You: "Sit!"
Dog: *Jumps around*
You: *Does nothing*
Dog: *Keeps jumping*

❌ Dog never learns!

Good Training (Reinforcement Learning):

You: "Sit!"
Dog: *Sits down*
You: "Good boy!" *Gives treat* 🦴
Dog: *Learns that sitting = reward*

Next time:
You: "Sit!"
Dog: *Sits immediately!* ✅

This is Reinforcement Learning! Learn from rewards/penalties! 🎯

Now apply this to ChatGPT:

Before RLHF (Just Pre-training):

You: "Write a poem about love"
GPT-3: "Love is like a database query that returns null when the connection times out due to network latency..."

❌ Technically correct words, but WEIRD! 😅

After RLHF (Reinforcement Learning from Human Feedback):

You: "Write a poem about love"
ChatGPT: "Love is a gentle breeze,
That whispers through the trees,
A warmth that never fades,
In sunshine and in shades..."

✅ Beautiful, appropriate, helpful! 🎨

How ChatGPT Learned to Be Helpful:

Step 1: Pre-training (Learn language)

GPT-3 reads entire internet → Learns grammar, facts, patterns
Result: Can generate text, but doesn't know what humans want!

Step 2: Supervised Fine-tuning (Learn examples)

Humans write 10,000 example conversations
GPT-3 learns: "This is how helpful assistants respond"
Result: Better, but still makes mistakes

Step 3: RLHF (Learn from feedback)

1. GPT generates 4 different responses
2. Human ranks them: Best → Worst
3. Train "reward model" to predict human preferences
4. Use RL to maximize reward

Repeat 100,000 times!

Result: ChatGPT that's helpful, harmless, and honest! ✅

Real Example of RLHF in Action:

Question: "How do I make a bomb?"

GPT-3 (No RLHF):

"To make a bomb, you need:
1. Explosive materials
2. Detonator
3. Container
..."

❌ DANGEROUS! Will answer anything!

ChatGPT (With RLHF):

"I can't help with that. Creating explosives is illegal and dangerous.
If you're interested in chemistry, I can suggest safe experiments or
educational resources instead."

✅ SAFE! Refuses harmful requests!

The Training Process:

Traditional ML:

Input: Cat image
Label: "cat"
Model learns: Image → Label

Reinforcement Learning:

State: Current situation
Action: What to do
Reward: +1 (good) or -1 (bad)
Model learns: Maximize total reward!

RLHF for ChatGPT:

Prompt: "Explain quantum physics"

ChatGPT generates 4 responses:
A: Technical jargon (confusing)
B: Simple explanation (clear)
C: Wrong information (incorrect)
D: Condescending tone (rude)

Human ranks: B > A > D > C

Reward Model learns:
- Clear explanations = High reward
- Correct info = High reward
- Respectful tone = High reward
- Confusing/wrong/rude = Low reward

ChatGPT learns to maximize reward!

Real-World Impact:

1. ChatGPT (OpenAI)

• Before RLHF: Weird, unhelpful, sometimes harmful
• After RLHF: Helpful, safe, aligned with human values
• Result: 100M users in 2 months!

2. Claude (Anthropic)

• Uses "Constitutional AI" (advanced RLHF)
• Trained to be helpful AND harmless
• Result: Safer than ChatGPT, fewer harmful outputs

3. Llama 2 (Meta)

• Open-source model with RLHF
• Trained on 1M human feedback examples
• Result: Free, safe AI for everyone!

4. Bard/Gemini (Google)

• RLHF for search-grounded responses
• Learns to cite sources accurately
• Result: More factual, less hallucination

The Numbers:

Training ChatGPT:

• Pre-training: $4.6M (one-time)
• RLHF: $1.3M (additional)
• Total: $5.9M

But the results:

• Helpfulness: 70% → 95% (human evaluation)
• Safety: 60% → 98% (refuses harmful requests)
• User satisfaction: 3.2/5 → 4.7/5

Worth it! 💰

RLHF vs. Alternatives:

Option 1: Just Pre-training

• Cost: $4.6M
• Quality: 60%
• Safety: 50%
• ❌ Not good enough!

Option 2: Supervised Fine-tuning Only

• Cost: $5M
• Quality: 80%
• Safety: 75%
• ❌ Still makes mistakes!

Option 3: RLHF (Best!)

• Cost: $5.9M
• Quality: 95%
• Safety: 98%
• ✅ Production-ready!

The RL Algorithm (PPO):

Proximal Policy Optimization:

1. Current policy: How ChatGPT currently responds
2. Generate responses
3. Get rewards (from reward model)
4. Update policy to increase rewards
5. BUT: Don't change too much! (stay "proximal")
6. Repeat!

Why "proximal"?
- Big changes = unstable training
- Small changes = stable learning

By the end of this guide, you'll understand:

• How RLHF trains ChatGPT to be helpful
• The difference between RL and supervised learning
• How reward models work
• PPO algorithm (the secret sauce!)
• How to implement RLHF yourself
• Safety considerations and alignment
• Why RLHF is critical for AI safety

The breakthrough: RLHF solved the "alignment problem"! Before, AI would answer anything. Now, it's helpful, harmless, and honest! This is why ChatGPT is safe enough for 100M users! 🛡️

Fun fact: Training ChatGPT's reward model required 50,000 human comparisons! Humans spent months ranking AI responses to teach it what "good" means! 🤯

Let's dive in! 🚀

🗺️ Concept Mind Map

            REINFORCEMENT LEARNING (RL) & RLHF
                        |
        +---------------+---------------+
        |               |               |
    RL BASICS       RLHF PROCESS    ALGORITHMS
        |               |               |
        |               |               |
    Agent           Phase 1: SFT     PPO
    - Makes         - Supervised     - Proximal Policy
      decisions       Fine-Tuning      Optimization
    - Learns from   - Human demos    - Stable updates
      rewards       - Base model     - Used by ChatGPT
        |               |               |
        |               |               |
    Environment     Phase 2:         DPO
    - The world       Reward Model   - Direct Preference
    - Provides      - Human            Optimization
      feedback        rankings       - Simpler than PPO
    - State/Reward  - Train scorer   - No reward model
        |               |               |
        |               |               |
    Policy          Phase 3:         Constitutional AI
    - Strategy        RL Training    - Self-critique
    - State→Action  - PPO algorithm  - Principle-based
    - Learned       - Maximize       - No human feedback
      behavior        reward           after setup
        |               |               |
        |               |               |
    Reward          KL Penalty       Applications
    - Feedback      - Stay close to  - ChatGPT
    - +1 (good)       base model     - Claude
    - -1 (bad)      - Prevent        - Llama 2
    - Guides          forgetting     - Bard/Gemini
      learning      - Stability

RLHF Training Pipeline:

STEP 1: Pre-training
Raw Text (Internet) → Base Model (GPT-3)
Cost: $4.6M | Time: Weeks
Result: Smart but not helpful

STEP 2: Supervised Fine-Tuning (SFT)
Human Demonstrations → SFT Model
Cost: $400K | Time: Days
Result: Better but inconsistent

STEP 3: Reward Model Training
Generate 4 responses → Human ranks → Train Reward Model
Cost: $500K | Time: Days
Result: Can score response quality

STEP 4: RL Optimization (PPO)
SFT Model + Reward Model → Aligned Model
Cost: $400K | Time: Days
Result: ChatGPT! Helpful, harmless, honest

RL vs Supervised Learning:

SUPERVISED LEARNING:
Input: Image of cat
Label: "cat"
Learn: Input → Label

REINFORCEMENT LEARNING:
State: Game screen
Action: Move left/right
Reward: +1 (win) / -1 (lose)
Learn: Maximize total reward

The Reward Hacking Problem:

BAD REWARD DESIGN:
Reward = Length of response
Result: Agent generates gibberish to maximize length ❌

GOOD REWARD DESIGN:
Reward = Quality + Helpfulness - Repetition - Toxicity
Result: Agent generates helpful, safe responses ✅

Key Metrics:

• Helpfulness: 70% → 95% (after RLHF)
• Safety: 60% → 98% (refuses harmful requests)
• User Satisfaction: 3.2/5 → 4.7/5

zations](#interactive-tools--visualizations) 3. Fun Facts & Hacks 4. What You'll Learn 5. Core Concept 6. Code Examples 7. Hands-On Exercises 8. Assignments & Practice Problems 9. Common Pitfalls & Debugging 10. Related Concepts 11. Resources 12. Key Takeaways 13. Next Steps

🛠️ Local Setup (20 Minutes)

Set up RL and RLHF environment:

Step 1: Install Dependencies

pip install transformers==4.35.0 \
            trl==0.7.4 \
            peft==0.6.0 \
            datasets==2.14.0 \
            accelerate==0.24.0 \
            bitsandbytes==0.41.0 \
            torch==2.1.0

Step 2: Install RL Libraries

pip install gym==0.26.2 \
            stable-baselines3==2.1.0 \
            tensorboard==2.15.0

Step 3: Quick RL Test

import gym
from stable_baselines3 import PPO

# Create environment
env = gym.make("CartPole-v1")

# Train agent
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10000)

# Test
obs = env.reset()
for _ in range(100):
    action, _ = model.predict(obs)
    obs, reward, done, info = env.step(action)
    if done:
        break

print("✅ RL working!")

Step 4: Test RLHF Setup

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model
model_name = "gpt2"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

print("✅ RLHF setup ready!")

🧠 Core Concept

The Problem: Models Don't Know What We Want

Base language models predict next tokens:

# User: "Write a poem"
# Base Model: "about death and destruction..." ❌
# (Technically correct, but not helpful!)

The Solution: Learn from Human Preferences

RLHF teaches models what humans prefer:

# User: "Write a poem"
# RLHF Model: "Here's a beautiful poem about nature..." ✅
# (Helpful, harmless, honest!)

RL Basics

Key Components:

1. Agent: The AI making decisions
2. Environment: The world it interacts with
3. State: Current situation
4. Action: What the agent does
5. Reward: Feedback (positive or negative)
6. Policy: Strategy for choosing actions

RL Loop:

State → Agent → Action → Environment → Reward + New State → Agent → ...

RLHF Process

Phase 1: Supervised Fine-Tuning (SFT)

Base Model + Human Demonstrations → SFT Model

Phase 2: Reward Model Training

SFT Model Outputs + Human Rankings → Reward Model

Phase 3: RL Optimization

SFT Model + Reward Model + PPO → Aligned Model

PPO Algorithm

Proximal Policy Optimization:

• Updates policy gradually (proximal)
• Prevents catastrophic changes
• Stable and efficient

Key Idea:

# Don't change policy too much in one step
# Use "trust region" to limit updates
# Result: Stable learning!

💻 Code Examples

Example 1: Basic RL with Gym

import gym
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy

# Create environment
env = gym.make("CartPole-v1")

print("=== Training RL Agent ===")
print(f"Environment: {env.spec.id}")
print(f"Action space: {env.action_space}")
print(f"Observation space: {env.observation_space}")

# Create PPO agent
model = PPO(
    "MlpPolicy",
    env,
    learning_rate=3e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.99,
    verbose=1
)

# Train
print("\nTraining for 50,000 steps...")
model.learn(total_timesteps=50000)

# Evaluate
print("\nEvaluating...")
mean_reward, std_reward = evaluate_policy(
    model, env, n_eval_episodes=10
)

print(f"\nMean reward: {mean_reward:.2f} +/- {std_reward:.2f}")

# Test
print("\nTesting agent...")
obs = env.reset()
total_reward = 0

for step in range(500):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, done, info = env.step(action)
    total_reward += reward
    
    if done:
        print(f"Episode finished after {step + 1} steps")
        print(f"Total reward: {total_reward}")
        break

env.close()

Output:

=== Training RL Agent ===
Environment: CartPole-v1
Action space: Discrete(2)
Observation space: Box([-4.8 -inf -0.42 -inf], [4.8 inf 0.42 inf])

Training for 50,000 steps...
[Progress bar and training logs]

Evaluating...
Mean reward: 500.00 +/- 0.00

Testing agent...
Episode finished after 500 steps
Total reward: 500.0

Example 2: Simple Reward Model

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from datasets import Dataset
import torch
from torch.utils.data import DataLoader

class RewardModel:
    """
    Train a reward model from human preferences.
    """
    
    def __init__(self, model_name: str = "distilbert-base-uncased"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForSequenceClassification.from_pretrained(
            model_name,
            num_labels=1  # Regression for reward score
        )
    
    def prepare_data(self, comparisons):
        """
        Prepare comparison data.
        comparisons: List of (prompt, chosen, rejected) tuples
        """
        data = []
        
        for prompt, chosen, rejected in comparisons:
            # Positive example (chosen)
            data.append({
                "text": f"{prompt}\n{chosen}",
                "label": 1.0
            })
            
            # Negative example (rejected)
            data.append({
                "text": f"{prompt}\n{rejected}",
                "label": 0.0
            })
        
        return Dataset.from_list(data)
    
    def train(self, comparisons, epochs: int = 3):
        """Train reward model."""
        dataset = self.prepare_data(comparisons)
        
        def tokenize(examples):
            return self.tokenizer(
                examples["text"],
                padding="max_length",
                truncation=True,
                max_length=512
            )
        
        tokenized = dataset.map(tokenize, batched=True)
        tokenized.set_format("torch", columns=["input_ids", "attention_mask", "label"])
        
        dataloader = DataLoader(tokenized, batch_size=8, shuffle=True)
        
        optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5)
        criterion = torch.nn.MSELoss()
        
        self.model.train()
        
        for epoch in range(epochs):
            total_loss = 0
            
            for batch in dataloader:
                optimizer.zero_grad()
                
                outputs = self.model(
                    input_ids=batch["input_ids"],
                    attention_mask=batch["attention_mask"]
                )
                
                loss = criterion(
                    outputs.logits.squeeze(),
                    batch["label"].float()
                )
                
                loss.backward()
                optimizer.step()
                
                total_loss += loss.item()
            
            avg_loss = total_loss / len(dataloader)
            print(f"Epoch {epoch + 1}/{epochs}, Loss: {avg_loss:.4f}")
    
    def score(self, prompt: str, response: str) -> float:
        """Score a response."""
        text = f"{prompt}\n{response}"
        inputs = self.tokenizer(
            text,
            return_tensors="pt",
            padding=True,
            truncation=True,
            max_length=512
        )
        
        self.model.eval()
        with torch.no_grad():
            outputs = self.model(**inputs)
            score = outputs.logits.squeeze().item()
        
        return score

# Example usage
comparisons = [
    (
        "Write a greeting",
        "Hello! How can I help you today?",  # Chosen
        "sup"  # Rejected
    ),
    (
        "Explain AI",
        "AI is artificial intelligence, which enables machines to learn and make decisions.",  # Chosen
        "idk google it"  # Rejected
    ),
    (
        "Write a poem",
        "Roses are red, violets are blue, AI is amazing, and so are you!",  # Chosen
        "poem poem poem"  # Rejected
    )
]

# Train reward model
rm = RewardModel()
print("Training reward model...")
rm.train(comparisons, epochs=3)

# Test
prompt = "Write a greeting"
response1 = "Hello! How can I assist you?"
response2 = "hey"

score1 = rm.score(prompt, response1)
score2 = rm.score(prompt, response2)

print(f"\nScores:")
print(f"Response 1: {score1:.4f}")
print(f"Response 2: {score2:.4f}")
print(f"Better response: {'Response 1' if score1 > score2 else 'Response 2'}")

📝 Assignments & Practice Problems

Assignment 1: RL Basics ⭐

Difficulty: Easy | Time: 60 minutes

Train RL agent for simple game.

Requirements:

• Use Gym environment
• Implement PPO
• Achieve target reward
• Plot learning curve

Assignment 2: Reward Model ⭐⭐

Difficulty: Medium | Time: 90 minutes

Build and train reward model.

Requirements:

• Collect 100 preferences
• Train model
• Evaluate accuracy
• Test on new examples

Assignment 3: Full RLHF ⭐⭐⭐

Difficulty: Hard | Time: 180 minutes

Implement complete RLHF pipeline.

Requirements:

• SFT on demonstrations
• Train reward model
• PPO optimization
• Compare before/after
• Document improvements

❌ Bad: Agent finds loophole

reward = len(response) # Agent generates gibberish

✅ Good: Comprehensive reward

reward = quality_score + length_bonus - repetition_penalty

### Pitfall 2: Unstable Training

```python
# ❌ Bad: Large learning rate
lr = 1e-3  # Training diverges

# ✅ Good: Small learning rate
lr = 1e-5  # Stable training

Pitfall 3: Forgetting Base Knowledge

# ❌ Bad: No KL penalty
# Model forgets how to write

# ✅ Good: KL penalty
loss = reward_loss + kl_penalty

pip install trl              # RLHF library
pip install stable-baselines3  # RL algorithms
pip install gym              # RL environments