zations|🎮 Interactive Tools & Visualizations]] - Local Visualization

• 🎉 Fun Facts & Hacks
  • 🤯 Mind-Blowing Facts
  • 💰 Performance Hacks
  • 🎯 Practical Hacks
• 🎯 What You'll Learn
• 🧠 Core Concept
  • The Problem with RNNs/LSTMs
  • The Transformer Solution
  • Key Innovation: "Attention Is All You Need"
• 💻 Code Examples
  • Example 1: Positional Encoding
  • Example 2: Feed-Forward Network
  • Example 3: Layer Normalization
  • Example 4: Complete Transformer Block
  • Example 5: Building a Mini-Transformer with PyTorch
• 🎯 Hands-On Exercises
  • Exercise 1: Implement Encoder-Decoder Attention
  • Exercise 2: Visualize Transformer Attention Patterns
• 🔍 Common Pitfalls & Debugging
  • Pitfall 1: Forgetting Positional Encoding
  • Pitfall 2: Incorrect Masking
  • Pitfall 3: Dimension Mismatches
• 🔗 Related Concepts
• 📚 Resources
  • Essential Papers
  • Video Tutorials
  • Interactive Tools
  • Code Libraries
• ✅ Key Takeaways
• 🎯 Next Steps

🤖 Transformer Architecture

🌟 Start With a Real-World Scenario

Imagine you're building an AI that can:

• Translate any language to any other language 🌍
• Write code in Python, JavaScript, and Rust 💻
• Answer questions about history, science, and math 📚
• Generate images from text descriptions 🎨
• Compose music in different styles 🎵

Sounds impossible, right? You'd need different AI models for each task!

But wait... ChatGPT, Claude, and GPT-4 do ALL of this with the same architecture: The Transformer! 🤯

The Problem Before Transformers (2017):

Old AI models (RNNs, LSTMs) had to process text one word at a time:

"The cat sat on the mat"
Step 1: Process "The"
Step 2: Process "cat" (remember "The")
Step 3: Process "sat" (remember "The cat")
Step 4: Process "on" (remember "The cat sat")
...

Issues:

• ❌ Slow (can't parallelize)
• ❌ Forgets early words in long sentences
• ❌ Can't see the full picture
• ❌ Hard to train on GPUs

The Transformer Revolution (2017):

Transformers process ALL words simultaneously:

"The cat sat on the mat"
↓
Process ALL words at once using attention!
↓
Understand relationships between ALL words instantly

Benefits:

• ✅ Fast (fully parallelizable on GPUs)
• ✅ Remembers everything (attention to all words)
• ✅ Sees the full context
• ✅ Scales to billions of parameters

Real-World Impact:

Before Transformers (2016):

• Google Translate: Okay-ish translations
• Chatbots: Very limited, scripted responses
• Code generation: Didn't exist
• Image generation: Separate, limited models

After Transformers (2017-2024):

• GPT-4: Writes essays, code, poems, jokes
• BERT: Powers Google Search (understands context)
• Claude: Long conversations with perfect memory
• DALL-E: Generates images from text
• Codex: GitHub Copilot writes code
• AlphaFold: Predicts protein structures
• Whisper: Transcribes speech in 100 languages

All using the same Transformer architecture! 🚀

🖼️ Master Architecture Diagram

Transformer: From Token Vectors to Contextual Understanding

Follow the 7 key stages: from tokenization of "What is the capital of India?", through 1,536D embedding matrix lookup, positional injection, Multi-Head Attention (Syntax & Entity Relations), FFN layer expansion ($1,536 \to 6,144 \to 1,536$), to the final contextualized token representations.

Infographic takeaway: what changed after attention

The transformer evolution visuals reinforce a simple progression:

N-grams
→ Neural language models
→ RNNs
→ LSTMs
→ Attention
→ Transformers
→ GPT / BERT / modern LLMs

The big shift was not just “bigger models.” It was a new way to process sequences:

• no more strict left-to-right recurrence
• direct token-to-token interaction through attention
• massive parallelism on GPUs
• much better scaling with data and compute

The "Aha!" Moment:

Think of Transformers like a team meeting:

Old way (RNN): Everyone speaks one at a time, in order

• Person 1 speaks → Person 2 speaks → Person 3 speaks
• By the time Person 10 speaks, everyone forgot what Person 1 said!

Transformer way: Everyone speaks simultaneously, and everyone listens to everyone

• All voices heard at once
• Attention mechanism decides who to listen to most
• No information lost!

By the end of this guide, you'll understand:

• How Transformers process entire sequences in parallel
• Why "Attention is All You Need" (the famous paper title)
• The architecture behind GPT, BERT, and all modern LLMs
• How to build a Transformer from scratch
• Why Transformers revolutionized not just NLP, but ALL of AI

The crazy part? The Transformer architecture is surprisingly simple! It's just:

1. Attention (you learned this!) 👁️
2. Feed-forward networks (basic neural networks)
3. Layer normalization (keeps numbers stable)
4. Residual connections (helps training)

That's it! This simple architecture powers ChatGPT, which has 175 billion parameters! 🤯

Let's dive in! 🚀

🎨 Visual Architecture

Complete Transformer Architecture

graph TB
    subgraph "Encoder (Left Side)"
        Input1[Input Embedding] --> PE1[+ Positional Encoding]
        PE1 --> MHA1[Multi-Head<br/>Self-Attention]
        MHA1 --> Add1[Add & Norm]
        PE1 -.Residual.-> Add1
        Add1 --> FFN1[Feed Forward<br/>Network]
        FFN1 --> Add2[Add & Norm]
        Add1 -.Residual.-> Add2
        Add2 --> EncOut[Encoder Output]
    end
    
    subgraph "Decoder (Right Side)"
        Input2[Output Embedding] --> PE2[+ Positional Encoding]
        PE2 --> MMHA[Masked Multi-Head<br/>Self-Attention]
        MMHA --> Add3[Add & Norm]
        PE2 -.Residual.-> Add3
        
        Add3 --> CrossAttn[Cross-Attention<br/>to Encoder]
        EncOut --> CrossAttn
        CrossAttn --> Add4[Add & Norm]
        Add3 -.Residual.-> Add4
        
        Add4 --> FFN2[Feed Forward<br/>Network]
        FFN2 --> Add5[Add & Norm]
        Add4 -.Residual.-> Add5
        Add5 --> Linear[Linear Layer]
        Linear --> Softmax[Softmax]
        Softmax --> Output[Output Probabilities]
    end
    
    style Input1 fill:#FF9800,color:#fff
    style Input2 fill:#FF9800,color:#fff
    style Output fill:#4CAF50,color:#fff
    style EncOut fill:#2196F3,color:#fff

Encoder vs Decoder vs Encoder-Decoder

graph LR
    subgraph "Encoder-Only (BERT)"
        E1[Input Text] --> E2[Bidirectional<br/>Attention]
        E2 --> E3[Understanding<br/>Context]
        E3 --> E4[Classification<br/>Q&A, NER]
    end
    
    subgraph "Decoder-Only (GPT)"
        D1[Input Text] --> D2[Causal<br/>Attention]
        D2 --> D3[Generate<br/>Next Token]
        D3 --> D4[Text Generation<br/>Completion]
    end
    
    subgraph "Encoder-Decoder (T5)"
        ED1[Source Text] --> ED2[Encoder]
        ED2 --> ED3[Decoder]
        ED3 --> ED4[Target Text<br/>Translation]
    end
    
    style E1 fill:#2196F3,color:#fff
    style D1 fill:#9C27B0,color:#fff
    style ED1 fill:#FF9800,color:#fff

Positional Encoding Visualization

graph TB
    Seq["Sequence:<br/>'The cat sat'"] --> Tokens[Token Embeddings<br/>512 dimensions]
    
    Tokens --> T1["'The'<br/>[0.2, -0.5, ...]"]
    Tokens --> T2["'cat'<br/>[0.8, 0.1, ...]"]
    Tokens --> T3["'sat'<br/>[-0.3, 0.7, ...]"]
    
    PE[Positional Encoding] --> P1["Position 0<br/>[sin/cos pattern]"]
    PE --> P2["Position 1<br/>[sin/cos pattern]"]
    PE --> P3["Position 2<br/>[sin/cos pattern]"]
    
    T1 --> Add1[+]
    P1 --> Add1
    Add1 --> Final1["Token + Position"]
    
    T2 --> Add2[+]
    P2 --> Add2
    Add2 --> Final2["Token + Position"]
    
    T3 --> Add3[+]
    P3 --> Add3
    Add3 --> Final3["Token + Position"]
    
    Final1 --> Trans[Transformer<br/>Processing]
    Final2 --> Trans
    Final3 --> Trans
    
    style Seq fill:#FF9800,color:#fff
    style Trans fill:#4CAF50,color:#fff

Information Flow in Transformer

sequenceDiagram
    participant Input as Input Text
    participant Embed as Embeddings
    participant Encoder as Encoder Stack
    participant Decoder as Decoder Stack
    participant Output as Output Text
    
    Input->>Embed: Tokenize & Embed
    Embed->>Encoder: + Positional Encoding
    
    loop N Encoder Layers
        Encoder->>Encoder: Self-Attention
        Encoder->>Encoder: Feed Forward
    end
    
    Encoder->>Decoder: Context Vectors
    
    loop N Decoder Layers
        Decoder->>Decoder: Masked Self-Attention
        Decoder->>Encoder: Cross-Attention
        Decoder->>Decoder: Feed Forward
    end
    
    Decoder->>Output: Generate Tokens

Attention Patterns Comparison

graph TB
    subgraph "Self-Attention (Bidirectional)"
        SA1["The"] -.-> SA1
        SA1 -.-> SA2["cat"]
        SA1 -.-> SA3["sat"]
        SA2 -.-> SA1
        SA2 -.-> SA2
        SA2 -.-> SA3
        SA3 -.-> SA1
        SA3 -.-> SA2
        SA3 -.-> SA3
    end
    
    subgraph "Masked Self-Attention (Causal)"
        MA1["The"] -.-> MA1
        MA2["cat"] -.-> MA1
        MA2 -.-> MA2
        MA3["sat"] -.-> MA1
        MA3 -.-> MA2
        MA3 -.-> MA3
    end
    
    style SA2 fill:#4CAF50,color:#fff
    style MA2 fill:#FF9800,color:#fff

Transformer Evolution Timeline

timeline
    title Transformer Evolution
    2017 : Original Transformer
         : "Attention Is All You Need"
         : Encoder-Decoder for translation
    2018 : BERT (Encoder-only)
         : Bidirectional understanding
         : Masked Language Modeling
    2018 : GPT-1 (Decoder-only)
         : Unidirectional generation
         : 117M parameters
    2019 : GPT-2 (1.5B params)
         : Better text generation
         : Zero-shot learning
    2020 : GPT-3 (175B params)
         : Few-shot learning
         : In-context learning
    2020 : T5 (Encoder-Decoder)
         : Text-to-Text framework
         : Unified approach
    2022 : ChatGPT
         : RLHF alignment
         : Conversational AI
    2023 : GPT-4
         : Multimodal
         : Advanced reasoning

Reading the transformer timeline correctly

This timeline is useful because it shows that “transformer” is not just one model.

It became a family of designs:

• Encoder-only models like BERT for understanding
• Decoder-only models like GPT for generation
• Encoder-decoder models like T5 for input-to-output tasks such as translation and summarization

That is why one core architecture could spread across search, chat, code generation, translation, speech, and multimodal AI.

🗺️ Concept Mind Map

                    TRANSFORMER ARCHITECTURE
                              |
        +---------------------+---------------------+
        |                     |                     |
    ENCODER                DECODER            KEY COMPONENTS
        |                     |                     |
        |                     |                     |
    Input Embedding       Output Embedding    Multi-Head Attention
    + Positional          + Positional        - 8-16 parallel heads
      Encoding              Encoding          - Query, Key, Value
        ↓                     ↓                - Scaled dot-product
        |                     |                     |
    N × Encoder Blocks    N × Decoder Blocks  Positional Encoding
    - Self-Attention      - Masked Self-Attn  - sin/cos functions
    - Feed-Forward        - Cross-Attention   - Adds position info
    - Layer Norm          - Feed-Forward      - No recurrence needed
    - Residual            - Layer Norm            |
        ↓                     ↓                     |
        |                     |               Feed-Forward Network
    Encoder Output        Linear + Softmax    - 2 linear layers
    (Context Vector)      (Probabilities)     - ReLU activation
                                              - Position-wise
                                                  |
                                                  |
                                             Layer Normalization
                                             - Stabilizes training
                                             - Applied after each
                                               sub-layer

Architecture Flow:

ENCODER SIDE:                    DECODER SIDE:
Input Text                       Output Text (shifted right)
    ↓                                ↓
Tokenize & Embed                 Tokenize & Embed
    ↓                                ↓
+ Positional Encoding            + Positional Encoding
    ↓                                ↓
Multi-Head Self-Attention        Masked Self-Attention
    ↓                                ↓
Add & Norm                       Add & Norm
    ↓                                ↓
Feed-Forward                     Cross-Attention (with Encoder)
    ↓                                ↓
Add & Norm                       Add & Norm
    ↓                                ↓
[Repeat N times]                 Feed-Forward
    ↓                                ↓
Context Vectors ─────────────→   Add & Norm
                                     ↓
                                 [Repeat N times]
                                     ↓
                                 Linear + Softmax
                                     ↓
                                 Output Probabilities

Key Innovations:

• Parallel Processing: No sequential dependency (unlike RNNs)
• Self-Attention: Every word attends to every other word
• Positional Encoding: Injects sequence order information
• Residual Connections: Helps gradient flow in deep networks

🛠️ Local Setup (5 Minutes)

Build transformers from scratch and experiment with pre-trained models:

Step 1: Create Virtual Environment

python -m venv venv-transformer
source venv-transformer/bin/activate  # macOS/Linux
# venv-transformer\Scripts\activate  # Windows

Step 2: Install Dependencies

# Core libraries
pip install torch==2.0.1 \
            transformers==4.30.0 \
            numpy==1.24.3 \
            matplotlib==3.7.1 \
            einops==0.6.1 \
            tensorboard==2.13.0

# For visualization
pip install graphviz==0.20.1 \
            torchviz==0.0.2 \
            bertviz==1.4.0

Step 3: Verify Installation

python -c "
import torch
from transformers import AutoModel, AutoTokenizer
print('✅ PyTorch:', torch.__version__)
print('✅ Transformers installed!')
print('✅ CUDA available:', torch.cuda.is_available())
"

Step 4: Download Models for Experimentation

from transformers import AutoModel, AutoTokenizer

# Download BERT (encoder-only)
bert_model = AutoModel.from_pretrained('bert-base-uncased')
bert_tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# Download GPT-2 (decoder-only)
gpt2_model = AutoModel.from_pretrained('gpt2')
gpt2_tokenizer = AutoTokenizer.from_pretrained('gpt2')

print("✅ Models cached for offline use!")

🚨 Troubleshooting

Issue: "CUDA out of memory"

# Use CPU or smaller models
device = torch.device('cpu')
model = AutoModel.from_pretrained('distilbert-base-uncased')  # Smaller!

Issue: "Slow model loading"

# Models are cached in ~/.cache/huggingface/
# First download is slow, subsequent loads are fast

Visualize transformer architecture

from torchviz import make_dot import torch

model = torch.nn.Transformer(d_model=512, nhead=8) src = torch.rand(10, 32, 512) # (seq_len, batch, d_model) tgt = torch.rand(20, 32, 512)

output = model(src, tgt) dot = make_dot(output, params=dict(model.named_parameters())) dot.render('transformer_architecture', format='png') print("✅ Architecture saved as transformer_architecture.png")

---

## 🧠 Core Concept

### The Problem with RNNs/LSTMs

Before transformers, sequence models processed tokens **sequentially**:

```python
# RNN/LSTM: Sequential processing
for token in sequence:
    hidden_state = process(token, previous_hidden_state)
    # Must wait for previous token to finish!

# Problems:
# 1. Slow (can't parallelize)
# 2. Vanishing gradients for long sequences
# 3. Limited context window

The Transformer Solution

Process all tokens in parallel using attention:

# Transformer: Parallel processing
all_outputs = transformer(all_tokens)  # Process everything at once!

# Benefits:
# 1. Fast (fully parallelizable)
# 2. No vanishing gradients
# 3. Unlimited context (in theory)
# 4. Better at capturing long-range dependencies

Key Innovation: "Attention Is All You Need"

The 2017 paper showed that attention alone (without RNNs) is sufficient for state-of-the-art results.

💻 Code Examples

Example 1: Positional Encoding

import numpy as np
import matplotlib.pyplot as plt

def positional_encoding(seq_len, d_model):
    """
    Generate positional encodings to give the model information about
    token positions (since attention has no inherent notion of order).
    
    PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
    PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
    
    Args:
        seq_len: Sequence length
        d_model: Model dimension
    
    Returns:
        Positional encoding matrix (seq_len, d_model)
    """
    # Initialize position and dimension indices
    position = np.arange(seq_len)[:, np.newaxis]  # (seq_len, 1)
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    
    # Create positional encoding matrix
    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(position * div_term)  # Even indices
    pe[:, 1::2] = np.cos(position * div_term)  # Odd indices
    
    return pe

# Generate positional encodings
seq_len = 100
d_model = 512
pe = positional_encoding(seq_len, d_model)

print(f"Positional Encoding Shape: {pe.shape}")
print(f"First position encoding (first 10 dims): {pe[0, :10]}")

# Visualize positional encodings
plt.figure(figsize=(12, 8))
plt.imshow(pe, cmap='RdBu', aspect='auto')
plt.colorbar(label='Encoding Value')
plt.xlabel('Dimension')
plt.ylabel('Position')
plt.title('Positional Encoding Visualization')
plt.tight_layout()
plt.savefig('positional_encoding.png', dpi=300, bbox_inches='tight')
print("Visualization saved as 'positional_encoding.png'")

# Why this works: Similar positions have similar encodings
print("\n=== Position Similarity ===")
from sklearn.metrics.pairwise import cosine_similarity
pos_0 = pe[0:1]
pos_1 = pe[1:2]
pos_10 = pe[10:11]
pos_50 = pe[50:51]

print(f"Similarity(pos 0, pos 1): {cosine_similarity(pos_0, pos_1)[0][0]:.4f}")
print(f"Similarity(pos 0, pos 10): {cosine_similarity(pos_0, pos_10)[0][0]:.4f}")
print(f"Similarity(pos 0, pos 50): {cosine_similarity(pos_0, pos_50)[0][0]:.4f}")
print("→ Nearby positions are more similar!")

Example 2: Feed-Forward Network

import numpy as np

class FeedForward:
    """
    Position-wise Feed-Forward Network.
    Applied to each position independently and identically.
    
    FFN(x) = max(0, xW1 + b1)W2 + b2
    """
    
    def __init__(self, d_model, d_ff, dropout=0.1):
        """
        Args:
            d_model: Model dimension (e.g., 512)
            d_ff: Feed-forward dimension (typically 4 * d_model = 2048)
            dropout: Dropout rate
        """
        self.d_model = d_model
        self.d_ff = d_ff
        
        # Initialize weights
        self.W1 = np.random.randn(d_model, d_ff) * np.sqrt(2.0 / d_model)
        self.b1 = np.zeros(d_ff)
        self.W2 = np.random.randn(d_ff, d_model) * np.sqrt(2.0 / d_ff)
        self.b2 = np.zeros(d_model)
    
    def forward(self, x):
        """
        Forward pass.
        
        Args:
            x: Input (batch_size, seq_len, d_model)
        
        Returns:
            Output (batch_size, seq_len, d_model)
        """
        # First linear transformation + ReLU
        hidden = np.maximum(0, np.dot(x, self.W1) + self.b1)
        
        # Second linear transformation
        output = np.dot(hidden, self.W2) + self.b2
        
        return output

# Example usage
batch_size = 2
seq_len = 10
d_model = 512
d_ff = 2048

# Random input
x = np.random.randn(batch_size, seq_len, d_model)

# Create and apply feed-forward network
ffn = FeedForward(d_model, d_ff)
output = ffn.forward(x)

print(f"Input shape: {x.shape}")
print(f"Output shape: {output.shape}")
print(f"Feed-forward dimension: {d_ff}")
print(f"\nNote: FFN is applied independently to each position")

Example 3: Layer Normalization

import numpy as np

def layer_norm(x, gamma, beta, eps=1e-5):
    """
    Layer Normalization: Normalize across features for each sample.
    
    Unlike Batch Norm (normalizes across batch), Layer Norm normalizes
    across features, making it suitable for variable-length sequences.
    
    Args:
        x: Input (batch_size, seq_len, d_model)
        gamma: Scale parameter (d_model,)
        beta: Shift parameter (d_model,)
        eps: Small constant for numerical stability
    
    Returns:
        Normalized output (batch_size, seq_len, d_model)
    """
    # Calculate mean and variance across the feature dimension
    mean = np.mean(x, axis=-1, keepdims=True)
    var = np.var(x, axis=-1, keepdims=True)
    
    # Normalize
    x_norm = (x - mean) / np.sqrt(var + eps)
    
    # Scale and shift
    output = gamma * x_norm + beta
    
    return output

# Example
batch_size = 2
seq_len = 5
d_model = 8

x = np.random.randn(batch_size, seq_len, d_model) * 10 + 5  # Random data
gamma = np.ones(d_model)
beta = np.zeros(d_model)

output = layer_norm(x, gamma, beta)

print("=== Layer Normalization ===")
print(f"Input mean: {np.mean(x):.4f}, std: {np.std(x):.4f}")
print(f"Output mean: {np.mean(output):.4f}, std: {np.std(output):.4f}")
print(f"\nPer-sample statistics (should be ~0 mean, ~1 std):")
for i in range(batch_size):
    for j in range(seq_len):
        sample = output[i, j]
        print(f"  Sample [{i},{j}]: mean={np.mean(sample):.4f}, std={np.std(sample):.4f}")

Example 4: Complete Transformer Block

import numpy as np

class TransformerBlock:
    """
    A single Transformer encoder block.
    
    Architecture:
    1. Multi-head self-attention
    2. Add & Norm (residual connection + layer norm)
    3. Feed-forward network
    4. Add & Norm
    """
    
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_ff = d_ff
        
        # Multi-head attention (simplified)
        self.attention_weights = np.random.randn(d_model, d_model) * 0.01
        
        # Feed-forward network
        self.W1 = np.random.randn(d_model, d_ff) * np.sqrt(2.0 / d_model)
        self.b1 = np.zeros(d_ff)
        self.W2 = np.random.randn(d_ff, d_model) * np.sqrt(2.0 / d_ff)
        self.b2 = np.zeros(d_model)
        
        # Layer norm parameters
        self.gamma1 = np.ones(d_model)
        self.beta1 = np.zeros(d_model)
        self.gamma2 = np.ones(d_model)
        self.beta2 = np.zeros(d_model)
    
    def layer_norm(self, x, gamma, beta, eps=1e-5):
        """Layer normalization."""
        mean = np.mean(x, axis=-1, keepdims=True)
        var = np.var(x, axis=-1, keepdims=True)
        x_norm = (x - mean) / np.sqrt(var + eps)
        return gamma * x_norm + beta
    
    def self_attention(self, x):
        """Simplified self-attention."""
        # In reality, this would be multi-head attention
        # For simplicity, we'll just do a linear transformation
        return np.dot(x, self.attention_weights)
    
    def feed_forward(self, x):
        """Feed-forward network."""
        hidden = np.maximum(0, np.dot(x, self.W1) + self.b1)
        return np.dot(hidden, self.W2) + self.b2
    
    def forward(self, x):
        """
        Forward pass through transformer block.
        
        Args:
            x: Input (batch_size, seq_len, d_model)
        
        Returns:
            Output (batch_size, seq_len, d_model)
        """
        # 1. Self-attention + residual + layer norm
        attn_output = self.self_attention(x)
        x = self.layer_norm(x + attn_output, self.gamma1, self.beta1)
        
        # 2. Feed-forward + residual + layer norm
        ff_output = self.feed_forward(x)
        x = self.layer_norm(x + ff_output, self.gamma2, self.beta2)
        
        return x

# Example usage
batch_size = 2
seq_len = 10
d_model = 512
num_heads = 8
d_ff = 2048

# Random input
x = np.random.randn(batch_size, seq_len, d_model)

# Create transformer block
block = TransformerBlock(d_model, num_heads, d_ff)

# Forward pass
output = block.forward(x)

print("=== Transformer Block ===")
print(f"Input shape: {x.shape}")
print(f"Output shape: {output.shape}")
print(f"Number of heads: {num_heads}")
print(f"Feed-forward dimension: {d_ff}")
print(f"\nKey components:")
print("  1. Multi-head self-attention")
print("  2. Residual connection + Layer norm")
print("  3. Feed-forward network")
print("  4. Residual connection + Layer norm")

Example 5: Building a Mini-Transformer with PyTorch

import torch
import torch.nn as nn
import math

class MiniTransformer(nn.Module):
    """
    A simplified but functional Transformer implementation.
    """
    
    def __init__(self, vocab_size, d_model=512, nhead=8, num_layers=6, 
                 d_ff=2048, max_seq_len=512, dropout=0.1):
        super().__init__()
        
        self.d_model = d_model
        
        # Token embeddings
        self.embedding = nn.Embedding(vocab_size, d_model)
        
        # Positional encoding
        self.pos_encoding = self.create_positional_encoding(max_seq_len, d_model)
        
        # Transformer encoder layers
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model,
            nhead=nhead,
            dim_feedforward=d_ff,
            dropout=dropout,
            batch_first=True
        )
        self.transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers)
        
        # Output projection
        self.output_projection = nn.Linear(d_model, vocab_size)
        
        self.dropout = nn.Dropout(dropout)
    
    def create_positional_encoding(self, max_len, d_model):
        """Create positional encoding matrix."""
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * 
                            (-math.log(10000.0) / d_model))
        
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        
        return pe.unsqueeze(0)  # Add batch dimension
    
    def forward(self, x, mask=None):
        """
        Forward pass.
        
        Args:
            x: Input token IDs (batch_size, seq_len)
            mask: Optional attention mask
        
        Returns:
            Output logits (batch_size, seq_len, vocab_size)
        """
        batch_size, seq_len = x.shape
        
        # 1. Token embeddings
        x = self.embedding(x) * math.sqrt(self.d_model)
        
        # 2. Add positional encoding
        x = x + self.pos_encoding[:, :seq_len, :].to(x.device)
        x = self.dropout(x)
        
        # 3. Transformer encoder
        x = self.transformer_encoder(x, mask=mask)
        
        # 4. Output projection
        logits = self.output_projection(x)
        
        return logits

# Example usage
vocab_size = 10000
batch_size = 4
seq_len = 20

# Create model
model = MiniTransformer(
    vocab_size=vocab_size,
    d_model=512,
    nhead=8,
    num_layers=6,
    d_ff=2048
)

# Random input (token IDs)
input_ids = torch.randint(0, vocab_size, (batch_size, seq_len))

# Forward pass
with torch.no_grad():
    output = model(input_ids)

print("=== Mini-Transformer ===")
print(f"Model parameters: {sum(p.numel() for p in model.parameters()):,}")
print(f"Input shape: {input_ids.shape}")
print(f"Output shape: {output.shape}")
print(f"\nArchitecture:")
print(f"  - Embedding dimension: 512")
print(f"  - Number of heads: 8")
print(f"  - Number of layers: 6")
print(f"  - Feed-forward dimension: 2048")
print(f"  - Vocabulary size: {vocab_size:,}")

# Predict next token
next_token_logits = output[:, -1, :]  # Last position
next_token_probs = torch.softmax(next_token_logits, dim=-1)
next_token = torch.argmax(next_token_probs, dim=-1)

print(f"\nNext token prediction: {next_token}")

Test with translation example

Encoder: "Le chat" (French)

Decoder: "The cat" (English)

### Exercise 2: Visualize Transformer Attention Patterns

```python
def visualize_transformer_layers(model, text, tokenizer):
    """
    Visualize attention patterns across all layers.
    
    Tasks:
    1. Extract attention weights from each layer
    2. Visualize how attention changes across layers
    3. Identify patterns (e.g., early layers focus on syntax,
       later layers focus on semantics)
    """
    # TODO: Implement visualization
    pass

❌ Without positional encoding

x = embedding(tokens) # Model has no notion of order!

✅ With positional encoding

x = embedding(tokens) + positional_encoding

### Pitfall 2: Incorrect Masking

```python
# For causal (autoregressive) models like GPT
def create_causal_mask(seq_len):
    """Prevent attending to future tokens."""
    mask = torch.triu(torch.ones(seq_len, seq_len), diagonal=1)
    mask = mask.masked_fill(mask == 1, float('-inf'))
    return mask

# For padding
def create_padding_mask(tokens, pad_token_id):
    """Mask out padding tokens."""
    return (tokens != pad_token_id).unsqueeze(1).unsqueeze(2)

Pitfall 3: Dimension Mismatches

# Common error: d_model not divisible by num_heads
d_model = 512
num_heads = 7  # ❌ 512 / 7 = 73.14... (not integer!)

# Fix: Use divisible numbers
num_heads = 8  # ✅ 512 / 8 = 64

<!--REF_CLOSE-->
# Install transformer libraries
pip install torch
pip install transformers
pip install einops  # For tensor operations
pip install flash-attn  # For efficient attention