🎓 Self-Supervised Learning
🌟 Start With a Real-World Scenario
Imagine teaching a child to read:
Traditional way (Supervised Learning):
- You: "This is a cat" 🐱 (show picture + label)
- You: "This is a dog" 🐕 (show picture + label)
- You: "This is a bird" 🐦 (show picture + label)
- Cost: You must label EVERY single image! 😰
Problem: To train GPT-4, you'd need to label:
- 13 trillion words
- Millions of images
- Thousands of hours of audio
- Cost: Billions of dollars! 💸
Impossible! 🚫
Self-Supervised Learning way:
Give the child a book with some words hidden:
"The cat sat on the ___"
Child learns to predict: "mat" ✅
"I love eating ___"
Child learns to predict: "pizza" / "ice cream" / "food" ✅
No labels needed! The text itself provides the supervision! 🎯
How GPT-4 Was Trained:
Step 1: Give it the entire internet (unlabeled!)
"The capital of France is ___"
Model learns: "Paris" ✅
"To print in Python, use ___"
Model learns: "print()" ✅
"E = mc___"
Model learns: "²" ✅
Step 2: Repeat billions of times
Result: Model learns:
- Grammar (from seeing correct sentences)
- Facts (from Wikipedia, books, articles)
- Reasoning (from math problems, code)
- Common sense (from everyday text)
All without a single human label! 🤯
Real-World Examples:
BERT (Google Search):
"The bank can refuse to lend money if the [MASK] account is empty"
BERT learns: [MASK] = "bank" (financial context)
GPT (ChatGPT):
"Once upon a time, there was a"
GPT learns to continue: "princess who lived in a castle..."
DALL-E (Image Generation):
Image: [Cat sitting on mat]
Text: "A cat sitting on a ___"
Learns: Images ↔ Text relationships
The Magic:
Traditional ML: Need 1 million labeled examples
- Cost: $100,000+ (humans labeling)
- Time: Months
- Limited to specific task
Self-Supervised Learning: Use 1 trillion unlabeled examples
- Cost: $0 for labels (just compute)
- Time: Weeks (parallel training)
- Works for ANY task!
By the end of this guide, you'll understand:
- How AI learns from unlabeled data (the internet!)
- Why GPT-4 didn't need humans to label 13 trillion words
- The difference between masked language modeling (BERT) and autoregressive modeling (GPT)
- How to train your own model on custom data without labeling
- Why self-supervised learning is called "the dark matter of intelligence"
The breakthrough: Self-supervised learning made large language models possible! Without it, we'd still be stuck with small, task-specific models that cost millions to train! 🚀
Fun fact: GPT-3 was trained on 45TB of text data. If you tried to label that manually at $0.01 per label, it would cost over $100 billion! Self-supervised learning made it possible for just $4.6 million in compute! 💰
Let's dive in! 🚀
🎨 Visual Understanding
Self-Supervised Learning Overview
graph TB
Data["Unlabeled Data<br/>(Internet, Books, Code)"] --> Create[Create Pretext Task]
Create --> MLM["Masked Language Modeling<br/>'The [MASK] sat on mat'"]
Create --> Next["Next Token Prediction<br/>'The cat' → predict 'sat'"]
Create --> Contrastive["Contrastive Learning<br/>Similar vs Different"]
MLM --> Train1[Train Model]
Next --> Train2[Train Model]
Contrastive --> Train3[Train Model]
Train1 --> Pretrained["Pre-trained Model<br/>(General Knowledge)"]
Train2 --> Pretrained
Contrastive --> Pretrained
Pretrained --> FineTune[Fine-tune for<br/>Specific Task]
FineTune --> App1[Classification]
FineTune --> App2[Translation]
FineTune --> App3[Q&A]
style Data fill:#FF9800,color:#fff
style Pretrained fill:#4CAF50,color:#fff
style FineTune fill:#2196F3,color:#fff
Supervised vs Self-Supervised Learning
graph LR
subgraph "Supervised Learning"
S1["Labeled Data<br/>Image + 'Cat'<br/>Cost: $$$"] --> S2[Train Model]
S2 --> S3["Task-Specific<br/>Model"]
S3 --> S4["Limited<br/>Generalization"]
end
subgraph "Self-Supervised Learning"
SS1["Unlabeled Data<br/>Just Images/Text<br/>Cost: $"] --> SS2[Create Labels<br/>Automatically]
SS2 --> SS3[Train Model]
SS3 --> SS4["General<br/>Representation"]
SS4 --> SS5["Transfer to<br/>Any Task"]
end
style S1 fill:#F44336,color:#fff
style SS1 fill:#4CAF50,color:#fff
style SS5 fill:#2196F3,color:#fff
BERT vs GPT Training Approaches
graph TB
subgraph "BERT (Masked Language Modeling)"
B1["Input:<br/>'The cat sat on the mat'"] --> B2["Mask Random Tokens<br/>'The [MASK] sat on the [MASK]'"]
B2 --> B3["Bidirectional Context<br/>Look left AND right"]
B3 --> B4["Predict: 'cat', 'mat'"]
B4 --> B5["Loss: Cross-Entropy"]
B5 --> B6["Update Weights"]
end
subgraph "GPT (Autoregressive/Causal)"
G1["Input:<br/>'The cat sat on the mat'"] --> G2["Predict Next Token<br/>'The' → 'cat'<br/>'The cat' → 'sat'"]
G2 --> G3["Unidirectional Context<br/>Only look left"]
G3 --> G4["Generate Sequence"]
G4 --> G5["Loss: Cross-Entropy"]
G5 --> G6["Update Weights"]
end
style B1 fill:#2196F3,color:#fff
style G1 fill:#9C27B0,color:#fff
style B4 fill:#4CAF50,color:#fff
style G4 fill:#4CAF50,color:#fff
Training Pipeline Flow
sequenceDiagram
participant Data as Raw Data
participant Pretext as Pretext Task
participant Model as Neural Network
participant Loss as Loss Function
participant Optimizer as Optimizer
Data->>Pretext: Create self-supervised task
Pretext->>Model: Input (with hidden info)
Model->>Model: Forward pass
Model->>Loss: Predictions
Pretext->>Loss: Ground truth (from data)
Loss->>Optimizer: Calculate gradient
Optimizer->>Model: Update weights
Note over Data,Optimizer: Repeat billions of times!
Model->>Model: Learns general representations
Pre-training vs Fine-tuning
graph LR
subgraph "Pre-training (Self-Supervised)"
PT1["Massive Unlabeled Data<br/>Billions of examples"] --> PT2[Self-Supervised Task]
PT2 --> PT3["General Model<br/>Understands language"]
end
subgraph "Fine-tuning (Supervised)"
FT1["Small Labeled Data<br/>Thousands of examples"] --> FT2[Specific Task]
PT3 --> FT2
FT2 --> FT3["Task-Specific Model<br/>Sentiment, QA, etc."]
end
style PT1 fill:#FF9800,color:#fff
style PT3 fill:#4CAF50,color:#fff
style FT3 fill:#2196F3,color:#fff
🗺️ Concept Mind Map
SELF-SUPERVISED LEARNING
|
+-----------------+-----------------+
| | |
PRETEXT TASKS APPROACHES APPLICATIONS
| | |
| | |
Masked Language BERT-Style Language Models
Modeling (MLM) - Bidirectional - BERT
- Mask 15% tokens - Predict masked - RoBERTa
- Predict masked - Example: - ALBERT
- BERT uses this "The [MASK] - DeBERTa
is blue"
| | |
| | |
Next Token GPT-Style Text Generation
Prediction - Unidirectional - GPT-2/3/4
- Predict next - Autoregressive - LLaMA
word - Example: - Mistral
- GPT uses this "The cat" → - Claude
predict "sat"
| | |
| | |
Contrastive SimCLR/MoCo Vision Models
Learning - Similar pairs - ResNet
- Positive pairs close together - ViT
(augmented) - Different pairs - CLIP
- Negative pairs far apart - DINO
(different)
| | |
| | |
Rotation Self-Distillation Multi-Modal
Prediction - Teacher-Student - CLIP
- Rotate image - Knowledge - ALIGN
- Predict angle transfer - Flamingo
- Vision task - No labels - GPT-4V
Learning Pipeline:
STEP 1: Pre-training (Self-Supervised)
Unlabeled Data (millions/billions)
↓
Create Pretext Task (auto-generate labels)
↓
Train Model on Pretext Task
↓
Learn General Representations
↓
STEP 2: Fine-tuning (Supervised)
Small Labeled Dataset (thousands)
↓
Fine-tune Pre-trained Model
↓
Task-Specific Model
Key Advantages:
- No Manual Labels: Uses unlabeled data (cheap & abundant)
- Transfer Learning: Pre-trained models work across tasks
- Better Performance: Often beats supervised learning
- Data Efficiency: Needs fewer labeled examples for fine-tuning
zations](#🎮-interactive-tools--visualizations) 3. Fun Facts & Hacks 4. What You'll Learn 5. Core Concept 6. Code Examples 7. Hands-On Exercises 8. Assignments & Practice Problems 9. Common Pitfalls & Debugging 10. Related Concepts 11. Resources 12. Key Takeaways 13. Next Steps
🛠️ Local Setup (5 Minutes)
Implement self-supervised learning from scratch:
Step 1: Create Virtual Environment
python -m venv venv-ssl
source venv-ssl/bin/activate # macOS/Linux
# venv-ssl\Scripts\activate # Windows
Step 2: Install Dependencies
pip install torch==2.0.1 \
transformers==4.30.0 \
datasets==2.12.0 \
numpy==1.24.3 \
matplotlib==3.7.1 \
tqdm==4.65.0 \
wandb==0.15.0 # For experiment tracking
Step 3: Verify Installation
python -c "
import torch
from transformers import AutoModelForMaskedLM
from datasets import load_dataset
print('✅ All libraries ready!')
print('✅ PyTorch:', torch.__version__)
"
Step 4: Download Sample Dataset
from datasets import load_dataset
# Download WikiText for practice
dataset = load_dataset('wikitext', 'wikitext-2-raw-v1')
print(f"✅ Dataset loaded: {len(dataset['train'])} examples")
Visualize masked token predictions
from transformers import pipeline
unmasker = pipeline('fill-mask', model='bert-base-uncased') results = unmasker("The cat [MASK] on the mat")
for result in results: print(f"{result['token_str']:>10} : {result['score']:.4f}")
Output:
sat : 0.8234
lay : 0.0543
sits : 0.0321
---
## 🧠 Core Concept
### The Problem: Labeled Data is Expensive
Traditional supervised learning requires labeled data:
```python
# Supervised learning
data = [
("The cat sat on the mat", "positive"), # Need human labels!
("I hate this movie", "negative"),
("Great product!", "positive"),
# ... millions more needed
]
# Problems:
# 1. Expensive to label (humans needed)
# 2. Limited scale (can't label entire internet)
# 3. Task-specific (labels only work for one task)
The Solution: Create Your Own Labels
Self-supervised learning creates labels from the data itself:
# Self-supervised learning
text = "The cat sat on the mat"
# Create supervision automatically:
# 1. Mask some words
masked = "The [MASK] sat on the [MASK]"
labels = ["cat", "mat"] # Predict masked words!
# 2. Predict next word
context = "The cat sat on the"
label = "mat" # Predict next word!
# No human labeling needed!
Key Insight: Pretext Tasks
A pretext task is a self-created task that forces the model to learn useful representations:
# Good pretext tasks:
# 1. Masked Language Modeling (BERT)
# 2. Next Token Prediction (GPT)
# 3. Contrastive Learning (SimCLR)
# 4. Rotation Prediction (images)
# 5. Jigsaw Puzzles (images)
# The model learns general features that transfer to real tasks!
💻 Code Examples
Example 1: Masked Language Modeling (BERT-style)
import numpy as np
import random
class MaskedLanguageModel:
"""
Implement Masked Language Modeling (MLM) pretext task.
Task: Predict masked tokens in a sentence.
Example: "The [MASK] sat on the mat" → predict "cat"
"""
def __init__(self, vocab, mask_prob=0.15):
self.vocab = vocab
self.mask_prob = mask_prob
self.mask_token = "[MASK]"
self.vocab_size = len(vocab)
# Add special tokens
if self.mask_token not in self.vocab:
self.vocab.append(self.mask_token)
self.token_to_id = {token: idx for idx, token in enumerate(self.vocab)}
self.id_to_token = {idx: token for token, idx in self.token_to_id.items()}
def create_mlm_data(self, text):
"""
Create masked language modeling training data.
BERT masking strategy:
- 80% of the time: replace with [MASK]
- 10% of the time: replace with random word
- 10% of the time: keep original word
Args:
text: Input text string
Returns:
masked_tokens: List of tokens with some masked
labels: Original tokens (for masked positions)
mask_positions: Positions that were masked
"""
tokens = text.split()
masked_tokens = tokens.copy()
labels = [-100] * len(tokens) # -100 = ignore in loss
mask_positions = []
for i, token in enumerate(tokens):
# Randomly decide whether to mask this token
if random.random() < self.mask_prob:
mask_positions.append(i)
labels[i] = self.token_to_id.get(token, 0)
# BERT masking strategy
rand = random.random()
if rand < 0.8:
# 80%: Replace with [MASK]
masked_tokens[i] = self.mask_token
elif rand < 0.9:
# 10%: Replace with random word
masked_tokens[i] = random.choice(self.vocab)
# else: 10%: Keep original (do nothing)
return masked_tokens, labels, mask_positions
def prepare_batch(self, texts):
"""Prepare a batch of MLM training data."""
batch_masked = []
batch_labels = []
batch_positions = []
for text in texts:
masked, labels, positions = self.create_mlm_data(text)
batch_masked.append(masked)
batch_labels.append(labels)
batch_positions.append(positions)
return batch_masked, batch_labels, batch_positions
# Example usage
vocab = ["the", "cat", "dog", "sat", "on", "mat", "a", "is", "animal"]
mlm = MaskedLanguageModel(vocab, mask_prob=0.15)
# Create training data
texts = [
"the cat sat on the mat",
"the dog is a animal",
"a cat is a animal"
]
print("=== Masked Language Modeling ===")
for text in texts:
masked, labels, positions = mlm.create_mlm_data(text)
print(f"\nOriginal: {text}")
print(f"Masked: {' '.join(masked)}")
print(f"Positions to predict: {positions}")
print(f"Labels: {[mlm.id_to_token.get(l, '[IGNORE]') for l in labels if l != -100]}")
# Prepare batch
batch_masked, batch_labels, batch_positions = mlm.prepare_batch(texts)
print(f"\n=== Batch Statistics ===")
print(f"Batch size: {len(batch_masked)}")
print(f"Average masks per sentence: {np.mean([len(p) for p in batch_positions]):.2f}")
Output:
=== Masked Language Modeling ===
Original: the cat sat on the mat
Masked: the [MASK] sat on the mat
Positions to predict: [1]
Labels: ['cat']
Original: the dog is a animal
Masked: the dog [MASK] a animal
Positions to predict: [2]
Labels: ['is']
Original: a cat is a animal
Masked: a cat is [MASK] animal
Positions to predict: [3]
Labels: ['a']
Example 2: Next Token Prediction (GPT-style)
import numpy as np
class NextTokenPrediction:
"""
Implement Next Token Prediction (autoregressive language modeling).
Task: Predict the next token given previous tokens.
Example: "The cat sat" → predict "on"
"""
def __init__(self, vocab):
self.vocab = vocab
self.token_to_id = {token: idx for idx, token in enumerate(vocab)}
self.id_to_token = {idx: token for token, idx in self.token_to_id.items()}
def create_training_pairs(self, text, context_length=3):
"""
Create (context, target) pairs for next token prediction.
Args:
text: Input text string
context_length: Number of previous tokens to use as context
Returns:
List of (context, target) pairs
"""
tokens = text.split()
pairs = []
for i in range(len(tokens) - 1):
# Get context (up to context_length previous tokens)
start = max(0, i - context_length + 1)
context = tokens[start:i+1]
target = tokens[i+1]
pairs.append((context, target))
return pairs
def create_causal_dataset(self, texts, max_length=10):
"""
Create dataset for causal language modeling.
Each example is a sequence where we predict each next token.
"""
dataset = []
for text in texts:
tokens = text.split()
# Create sequences of increasing length
for i in range(1, min(len(tokens), max_length)):
input_seq = tokens[:i]
target = tokens[i]
dataset.append((input_seq, target))
return dataset
# Example usage
vocab = ["the", "cat", "dog", "sat", "on", "mat", "a", "is", "animal", "quick", "brown", "fox"]
ntp = NextTokenPrediction(vocab)
# Create training pairs
text = "the quick brown fox sat on the mat"
pairs = ntp.create_training_pairs(text, context_length=3)
print("=== Next Token Prediction ===")
print(f"Text: {text}\n")
print("Training pairs (context → target):")
for context, target in pairs[:5]:
print(f" {' '.join(context):20} → {target}")
# Create causal dataset
texts = [
"the cat sat on the mat",
"the dog is a animal"
]
dataset = ntp.create_causal_dataset(texts, max_length=6)
print(f"\n=== Causal Dataset ===")
print(f"Total examples: {len(dataset)}")
print("\nFirst 5 examples:")
for input_seq, target in dataset[:5]:
print(f" Input: {' '.join(input_seq):20} → Target: {target}")
Output:
=== Next Token Prediction ===
Text: the quick brown fox sat on the mat
Training pairs (context → target):
the → quick
the quick → brown
quick brown → fox
brown fox → sat
fox sat → on
=== Causal Dataset ===
Total examples: 11
First 5 examples:
Input: the → Target: cat
Input: the cat → Target: sat
Input: the cat sat → Target: on
Input: the cat sat on → Target: the
Input: the cat sat on the → Target: mat
Example 3: Contrastive Learning
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
class ContrastiveLearning:
"""
Implement contrastive learning for text.
Idea: Similar texts should have similar embeddings,
dissimilar texts should have different embeddings.
"""
def __init__(self, embedding_dim=128, temperature=0.07):
self.embedding_dim = embedding_dim
self.temperature = temperature
def create_positive_pairs(self, text):
"""
Create positive pairs through data augmentation.
Augmentations for text:
1. Random word deletion
2. Random word swap
3. Synonym replacement
4. Back-translation
"""
tokens = text.split()
# Augmentation 1: Random deletion (keep 80% of words)
aug1_tokens = [t for t in tokens if np.random.random() > 0.2]
aug1 = ' '.join(aug1_tokens) if aug1_tokens else text
# Augmentation 2: Random swap
aug2_tokens = tokens.copy()
if len(aug2_tokens) > 1:
i, j = np.random.choice(len(aug2_tokens), 2, replace=False)
aug2_tokens[i], aug2_tokens[j] = aug2_tokens[j], aug2_tokens[i]
aug2 = ' '.join(aug2_tokens)
return aug1, aug2
def contrastive_loss(self, anchor, positive, negatives):
"""
Calculate contrastive loss (InfoNCE).
Loss = -log(exp(sim(anchor, positive) / temp) /
sum(exp(sim(anchor, all) / temp)))
Args:
anchor: Anchor embedding (embedding_dim,)
positive: Positive example embedding (embedding_dim,)
negatives: Negative examples (num_negatives, embedding_dim)
Returns:
Contrastive loss value
"""
# Calculate similarities
pos_sim = np.dot(anchor, positive) / (
np.linalg.norm(anchor) * np.linalg.norm(positive)
)
neg_sims = []
for neg in negatives:
sim = np.dot(anchor, neg) / (
np.linalg.norm(anchor) * np.linalg.norm(neg)
)
neg_sims.append(sim)
# Apply temperature scaling
pos_sim = pos_sim / self.temperature
neg_sims = np.array(neg_sims) / self.temperature
# InfoNCE loss
numerator = np.exp(pos_sim)
denominator = numerator + np.sum(np.exp(neg_sims))
loss = -np.log(numerator / denominator)
return loss
# Example usage
cl = ContrastiveLearning(embedding_dim=128, temperature=0.07)
# Create positive pairs
text = "the cat sat on the mat"
aug1, aug2 = cl.create_positive_pairs(text)
print("=== Contrastive Learning ===")
print(f"Original: {text}")
print(f"Augmentation 1: {aug1}")
print(f"Augmentation 2: {aug2}")
# Simulate embeddings (in practice, these come from the model)
np.random.seed(42)
anchor_emb = np.random.randn(128)
positive_emb = anchor_emb + np.random.randn(128) * 0.1 # Similar to anchor
negative_embs = np.random.randn(5, 128) # Different from anchor
# Calculate loss
loss = cl.contrastive_loss(anchor_emb, positive_emb, negative_embs)
print(f"\n=== Loss Calculation ===")
print(f"Contrastive loss: {loss:.4f}")
print(f"\nInterpretation:")
print(f" - Lower loss = positive pair is more similar than negatives")
print(f" - Model learns to pull similar texts together")
print(f" - And push dissimilar texts apart")
Example 4: Pre-training a Simple Language Model
import torch
import torch.nn as nn
import torch.optim as optim
class SimpleLM(nn.Module):
"""
Simple language model for demonstration.
"""
def __init__(self, vocab_size, embedding_dim=128, hidden_dim=256):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embedding_dim)
self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)
self.fc = nn.Linear(hidden_dim, vocab_size)
def forward(self, x):
# x: (batch_size, seq_len)
embedded = self.embedding(x) # (batch, seq_len, embedding_dim)
lstm_out, _ = self.lstm(embedded) # (batch, seq_len, hidden_dim)
logits = self.fc(lstm_out) # (batch, seq_len, vocab_size)
return logits
def pretrain_language_model(model, train_data, epochs=10, lr=0.001):
"""
Pre-train language model with next token prediction.
Args:
model: Language model
train_data: List of (input_ids, target_ids) tuples
epochs: Number of training epochs
lr: Learning rate
"""
optimizer = optim.Adam(model.parameters(), lr=lr)
criterion = nn.CrossEntropyLoss()
model.train()
for epoch in range(epochs):
total_loss = 0
for input_ids, target_ids in train_data:
# Forward pass
logits = model(input_ids)
# Reshape for loss calculation
# logits: (batch, seq_len, vocab_size)
# targets: (batch, seq_len)
loss = criterion(
logits.view(-1, logits.size(-1)),
target_ids.view(-1)
)
# Backward pass
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
avg_loss = total_loss / len(train_data)
if epoch % 2 == 0:
print(f"Epoch {epoch}, Loss: {avg_loss:.4f}")
return model
# Example usage
vocab_size = 1000
batch_size = 4
seq_len = 10
# Create model
model = SimpleLM(vocab_size, embedding_dim=128, hidden_dim=256)
# Create dummy training data
train_data = []
for _ in range(20):
input_ids = torch.randint(0, vocab_size, (batch_size, seq_len))
target_ids = torch.randint(0, vocab_size, (batch_size, seq_len))
train_data.append((input_ids, target_ids))
print("=== Pre-training Language Model ===")
print(f"Model parameters: {sum(p.numel() for p in model.parameters()):,}")
print(f"Vocabulary size: {vocab_size}")
print(f"Training examples: {len(train_data)}")
print("\nTraining...")
# Pre-train
model = pretrain_language_model(model, train_data, epochs=10, lr=0.001)
print("\n✓ Pre-training complete!")
print("Model has learned general language patterns")
print("Ready for fine-tuning on specific tasks")
Example 5: Fine-tuning Pre-trained Model
def finetune_for_classification(pretrained_model, train_data, num_classes, epochs=5):
"""
Fine-tune pre-trained model for classification.
Args:
pretrained_model: Pre-trained language model
train_data: List of (text, label) tuples
num_classes: Number of classification classes
epochs: Number of fine-tuning epochs
"""
# Freeze early layers (optional)
for param in pretrained_model.embedding.parameters():
param.requires_grad = False
# Add classification head
class ClassificationModel(nn.Module):
def __init__(self, base_model, hidden_dim, num_classes):
super().__init__()
self.base_model = base_model
self.classifier = nn.Linear(hidden_dim, num_classes)
def forward(self, x):
# Get representations from base model
logits = self.base_model(x)
# Use last token representation
last_hidden = logits[:, -1, :]
# Classify
return self.classifier(last_hidden)
model = ClassificationModel(pretrained_model, hidden_dim=256, num_classes=num_classes)
optimizer = optim.Adam(model.parameters(), lr=0.0001) # Lower LR for fine-tuning
criterion = nn.CrossEntropyLoss()
model.train()
for epoch in range(epochs):
total_loss = 0
correct = 0
total = 0
for input_ids, labels in train_data:
# Forward pass
outputs = model(input_ids)
loss = criterion(outputs, labels)
# Backward pass
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Calculate accuracy
_, predicted = torch.max(outputs, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
total_loss += loss.item()
accuracy = 100 * correct / total
avg_loss = total_loss / len(train_data)
print(f"Epoch {epoch}, Loss: {avg_loss:.4f}, Accuracy: {accuracy:.2f}%")
return model
# Example: Fine-tune for sentiment analysis
print("\n=== Fine-tuning for Sentiment Analysis ===")
# Create dummy fine-tuning data
num_classes = 2 # Positive/Negative
finetune_data = []
for _ in range(10):
input_ids = torch.randint(0, vocab_size, (batch_size, seq_len))
labels = torch.randint(0, num_classes, (batch_size,))
finetune_data.append((input_ids, labels))
print(f"Fine-tuning examples: {len(finetune_data)}")
print(f"Number of classes: {num_classes}")
print("\nFine-tuning...")
# Fine-tune
finetuned_model = finetune_for_classification(
model, finetune_data, num_classes, epochs=5
)
print("\n✓ Fine-tuning complete!")
print("Model adapted for sentiment classification")
❌ Fine-tuning with high learning rate
optimizer = optim.Adam(model.parameters(), lr=0.01) # Too high!
Model forgets pre-trained knowledge
✅ Use lower learning rate for fine-tuning
optimizer = optim.Adam(model.parameters(), lr=0.0001)
Preserves pre-trained knowledge while adapting
### Pitfall 2: Not Enough Pre-training Data
```python
# ❌ Pre-training on small dataset
texts = ["short text"] * 100 # Not enough diversity!
# ✅ Use large, diverse corpus
texts = load_large_corpus() # Millions of examples
# More data = better representations
Pitfall 3: Wrong Masking Strategy
# ❌ Always mask the same positions
masked_text = text.replace("the", "[MASK]") # Too predictable!
# ✅ Random masking
masked_text = randomly_mask(text, mask_prob=0.15)
# Model learns robust representations
🔗 Related Concepts
- Previous: Transformer - The architecture to train
- Next: LLMs - Self-supervised learning at scale
- Related: Fine-tuning - Adapting pre-trained models
- Advanced: RL - Learning from feedback
📚 Resources
Essential Papers
- 📄 BERT: Pre-training of Deep Bidirectional Transformers
- 📄 Language Models are Few-Shot Learners (GPT-3)
- 📄 A Simple Framework for Contrastive Learning (SimCLR)
- 📄 Self-Supervised Learning: A Survey
Video Tutorials
Interactive Tools
Code Libraries
<!--REF_CLOSE-->
# Install libraries for self-supervised learning
pip install transformers
pip install datasets
pip install torch
pip install wandb # For experiment tracking