🔨 BUILD: Simple Language Models

Learn by Building

Build two traditional language models from scratch and see their limitations firsthand.

✅ Code Available! All code examples are ready in the Code-Examples/ directory.

• 📁 N-gram Code
• 📁 LSTM Code

🎯 What You'll Build

Model 1: N-gram Language Model (20 minutes)

A simple statistical model that:
• Counts word sequences
• Predicts next word based on frequency
• Shows why traditional approaches fail

Model 2: LSTM Language Model (40 minutes)

A neural network that:
• Learns patterns from data
• Handles longer sequences
• Still has significant limitations

Why Build These?

✓ Understand how traditional LMs work
✓ See their limitations in practice
✓ Appreciate why we needed transformers
✓ Build foundation for understanding LLMs

⚙️ Setup

Prerequisites

Knowledge:

• Basic Python
• Understanding of probabilities (helpful but not required)
• Read 01 LM vs LLM Deep Dive first

Software:

# Python 3.8+
python --version

# Install dependencies
pip install torch numpy matplotlib

Project Structure

00-Kickstart/Code-Examples/
├── 01-ngram-language-model/
│   ├── README.md
│   ├── ngram_model.py
│   ├── train.py
│   └── sample_text.txt
└── 02-lstm-language-model/
    ├── README.md
    ├── lstm_model.py
    ├── train.py
    └── sample_text.txt

Part 1: N-gram Language Model

📖 Theory (5 minutes)

What is an N-gram?

N-gram = sequence of N words

Examples:
• Unigram (1-gram): "cat"
• Bigram (2-gram): "the cat"
• Trigram (3-gram): "the cat sat"

How N-gram Models Work:

# Training
Text: "I love cats. I love dogs."

# Count bigrams
("I", "love") → 2 times
("love", "cats") → 1 time
("love", "dogs") → 1 time

# Calculate probabilities
P("love" | "I") = 2/2 = 100%
P("cats" | "love") = 1/2 = 50%
P("dogs" | "love") = 1/2 = 50%

# Prediction
Input: "I"
Output: "love" (100% probability)

Input: "I love"
Output: "cats" or "dogs" (50% each)

🔨 Build It

Step 1: Navigate to code directory

cd 00-Kickstart/Code-Examples/01-ngram-language-model

Step 2: Review the code

Open ngram_model.py and read through it. Key concepts:

class NgramModel:
    def __init__(self, n=2):
        self.n = n  # Bigram by default
        self.ngrams = {}  # Store counts
    
    def train(self, text):
        # Count all n-grams
        pass
    
    def predict(self, context):
        # Return most likely next word
        pass

Step 3: Run training

python train.py

Expected Output:

Training N-gram model...
Vocabulary size: 150 words
Total bigrams: 500
Training complete!

Testing predictions:
Input: "the cat"
Output: "sat" (probability: 0.75)

Input: "I love"
Output: "cats" (probability: 0.50)

🧪 Experiment

Try these modifications in train.py:

# 1. Change N-gram size
model = NgramModel(n=3)  # Try trigrams

# 2. Add your own text
text = """
Your custom text here.
The model will learn from this.
"""

# 3. Test different inputs
predictions = model.predict("your input")

❌ Observe the Limitations

# Test these inputs and see failures:

# 1. Unseen combinations
model.predict("the dog")  # Never seen "dog" after "the"
# Output: None or random

# 2. Long-range dependencies
model.predict("the cat that I saw yesterday")
# Only looks at last 2 words, forgets "cat"

# 3. No understanding
model.predict("the happy")
# Might predict "cat" even though "happy cat" makes no sense

Full code reference: 📁 N-gram Code

Part 2: LSTM Language Model

📖 Theory (10 minutes)

What is an LSTM?

LSTM = Long Short-Term Memory
A type of neural network that:
• Remembers important information
• Forgets irrelevant details
• Handles sequences better than N-grams

Key Improvement over N-grams:

N-gram:
"the cat" → "sat"
(only looks at last 2 words)

LSTM:
"the cat that I saw yesterday" → "sat"
(can remember "cat" from earlier)

Architecture:

Input: "the cat sat"
       ↓
   [Embedding Layer]  ← Converts words to vectors
       ↓
   [LSTM Layers]      ← Learns patterns
       ↓
   [Output Layer]     ← Predicts next word
       ↓
Output: "on" (most likely)

🔨 Build It

Step 1: Navigate to code directory

cd 00-Kickstart/Code-Examples/02-lstm-language-model

Step 2: Understand the architecture

Open lstm_model.py:

class LSTMLanguageModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim=128, hidden_dim=256):
        super().__init__()
        # Word embeddings
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        
        # LSTM layers
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=2)
        
        # Output layer
        self.fc = nn.Linear(hidden_dim, vocab_size)
    
    def forward(self, x):
        # Convert words to embeddings
        embedded = self.embedding(x)
        
        # Process through LSTM
        output, (hidden, cell) = self.lstm(embedded)
        
        # Predict next word
        predictions = self.fc(output)
        return predictions

Step 3: Train the model

python train.py

Expected Output:

Training LSTM Language Model...
Epoch 1/10: Loss = 4.52
Epoch 2/10: Loss = 3.87
Epoch 3/10: Loss = 3.21
...
Epoch 10/10: Loss = 1.45
Training complete!

Model size: 2.5M parameters
Training time: 2 minutes

Testing:
Input: "the cat"
Output: "sat on the mat"

Input: "I love"
Output: "cats and dogs"

🧪 Experiment

Try these modifications:

# 1. Adjust model size
model = LSTMLanguageModel(
    vocab_size=vocab_size,
    embedding_dim=256,  # Bigger embeddings
    hidden_dim=512      # More LSTM units
)

# 2. Change training parameters
train(
    model,
    epochs=20,          # Train longer
    learning_rate=0.01  # Faster learning
)

# 3. Generate longer sequences
generated = model.generate(
    start_text="the cat",
    length=50  # Generate 50 words
)

✅ Observe the Improvements

# Test these and compare with N-gram:

# 1. Better with unseen combinations
model.predict("the dog")
# Output: "sat" (learned pattern, not just memorization)

# 2. Longer context
model.predict("the cat that I saw yesterday")
# Output: "was" (remembers "cat" from earlier)

# 3. More coherent
model.generate("once upon a time")
# Output: "once upon a time there was a cat that lived in a house"
# (more coherent than N-gram)

❌ But Still Limited

# These still fail:

# 1. Short memory (20-50 words)
long_text = "The cat... (40 words later) ... the"
model.predict(long_text)
# Forgets "cat" was mentioned

# 2. No real understanding
model.predict("the happy")
# Might still predict nonsense

# 3. Can't follow instructions
model.predict("translate to French: hello")
# No idea what "translate" means

Full code reference: 📁 LSTM Code

🔬 Compare the Models

Side-by-Side Comparison

Run the Comparison Script

cd 00-Kickstart/Code-Examples
python compare_models.py

Output:

Testing: "the cat sat on the"

N-gram Model:
→ "mat" (memorized from training)

LSTM Model:
→ "mat and looked around" (learned pattern)

What we need:
→ "mat, purring contentedly as it watched the birds outside"
   (understanding + creativity)

Try these challenges:

1. Train on different text

Use your own dataset (books, articles, code)

2. Improve the LSTM

Add more layers, bigger hidden size

3. Compare performance

Measure perplexity, generation quality

4. Visualize learning

Plot loss curves, attention weights

<!--REF_OPEN:Resources-->

- 📖 [Understanding LSTMs](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)
- 📖 [PyTorch LSTM Tutorial](https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html)
- 📄 [The Unreasonable Effectiveness of RNNs](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)

---

**Ready to build a real LLM?** → [🚀 BUILD: Mini Transformer LLM](00-Kickstart--01.b-BUILD-Mini-LLM.html)

---


<!--REF_CLOSE-->