concept_number: 19 title: Model Quantization stage: Advanced difficulty: ⭐⭐⭐⭐☆ time_to_learn: 4-5 hours status: not-started date_created: 2026-03-07 date_started: date_completed: progress: 0% prerequisites:

• "Large Language Models"
• "Knowledge Distillation"
• "Small Language Models" next_concepts:
• "Agents"
• "Small Language Models" related_concepts:
• "Knowledge Distillation"
• "Fine-Tuning" tags:
• ai
• quantization
• compression
• optimization
• int8
• int4

🗺️ Concept Mind Map

                QUANTIZATION
                        |
        +---------------+---------------+
        |               |               |
    PRECISION       TECHNIQUES      BENEFITS
        |               |               |
        |               |               |
    FP32 (32-bit)   Post-Training    Size Reduction
    - Full precision  Quantization   - FP32: 280GB
    - 4 bytes/param - After training - INT8: 70GB (4x)
    - Highest         - No retraining- INT4: 35GB (8x)
      quality       - Quick & easy   - Fits on GPU
        |               |               |
        |               |               |
    FP16 (16-bit)   Quantization-    Speed Boost
    - Half precision  Aware Training - INT8: 2-4x faster
    - 2 bytes/param - During training- Hardware support
    - 99% quality   - Better quality - Optimized ops
    - Common        - More work      - Real-time
        |               |               |
        |               |               |
    INT8 (8-bit)    Dynamic          Cost Savings
    - Integer         Quantization   - 8x less memory
    - 1 byte/param  - Runtime        - Cheaper hardware
    - 98% quality     conversion     - $80K → $1.5K
    - 2x smaller    - Flexible       - Accessible
        |               |               |
        |               |               |
    INT4 (4-bit)    Mixed Precision  Deployment
    - 4-bit integer - Critical layers- Edge devices
    - 0.5 byte/param  FP16           - Mobile phones
    - 95% quality   - Others INT8    - IoT devices
    - 4x smaller    - Best balance   - Laptops

Precision Comparison:

TYPE    SIZE/PARAM   QUALITY   SPEED    USE CASE
FP32    4 bytes      100%      1x       Research
FP16    2 bytes      99.9%     1.5x     Training
INT8    1 byte       98%       2-4x     Inference
INT4    0.5 bytes    95%       4-8x     Edge devices

Quantization Process:

STEP 1: Analyze Weight Distribution
Weights: [-2.5, -1.2, 0.3, 1.8, 2.1]
Range: -2.5 to 2.1

STEP 2: Map to Integer Range
FP32 → INT8 (256 values: -128 to 127)
Scale factor: 2.5 / 128 = 0.0195

STEP 3: Quantize
-2.5 → -128
-1.2 → -62
 0.3 → 15
 1.8 → 92
 2.1 → 108

STEP 4: Store as INT8
Memory: 5 bytes (was 20 bytes)
Savings: 75%!

Quality vs Size Trade-off:

MODEL SIZE    QUALITY   DEPLOYMENT
280GB (FP32)  100%      8× A100 GPUs ($80K)
140GB (FP16)  99.9%     4× A100 GPUs ($40K)
70GB (INT8)   98%       1× A100 GPU ($10K)
35GB (INT4)   95%       Consumer GPU ($1.5K)

Key Techniques:

• PTQ: Post-Training Quantization (fast, easy)
• QAT: Quantization-Aware Training (better quality)
• Dynamic: Runtime quantization (flexible)
• Mixed Precision: Different layers, different precision

🌟 Start With a Real-World Scenario

You want to run Llama-2-70B:

Full Precision (FP32):

Model size: 280GB (70B params × 4 bytes)
Memory needed: 8× A100 GPUs (80GB each)
Cost: $80,000 for hardware
Speed: 50 tokens/second

❌ Too expensive! Most people can't afford this!

Quantized (INT4):

Model size: 35GB (70B params × 0.5 bytes)
Memory needed: 1× Consumer GPU (48GB)
Cost: $1,500 for hardware
Speed: 50 tokens/second (same!)
Quality: 98% of original

✅ 8x smaller, 50x cheaper, same speed, minimal quality loss!

The Magic of Quantization:

What is it?

Full Precision (32-bit):
Number: 3.14159265359...
Storage: 4 bytes
Precision: Very high

Quantized (4-bit):
Number: 3.1
Storage: 0.5 bytes
Precision: Lower, but good enough!

Result: 8x smaller, minimal accuracy loss!

Real Example:

Original Model (FP16):

Weight: 0.847362518
Storage: 2 bytes

Quantized (INT8):

Weight: 0.85 (rounded)
Storage: 1 byte
Loss: 0.3% error

Quantized (INT4):

Weight: 0.8 (more rounding)
Storage: 0.5 bytes
Loss: 5% error

For 70B parameters:

• FP16: 140GB
• INT8: 70GB (2x smaller!)
• INT4: 35GB (4x smaller!)

The Surprise:

Quality Loss:

• INT8: 0-1% accuracy loss (imperceptible!)
• INT4: 2-5% accuracy loss (barely noticeable!)
• INT3: 5-10% accuracy loss (noticeable)
• INT2: 15-30% accuracy loss (significant)

Sweet spot: INT4!

• 4x smaller
• 4x faster
• 2-5% quality loss
• Runs on consumer hardware!

Real-World Impact:

1. Llama.cpp (Open Source)

Before: Llama-2-70B needs $80K GPU cluster
After: Runs on MacBook Pro M2!
Method: GGUF quantization (INT4)
Result: AI for everyone!

2. GPTQ (Quantization Method)

Before: 70B model = 140GB
After: 70B model = 35GB
Quality: 98% preserved
Speed: 4x faster!

3. AWQ (Activation-aware Quantization)

Smarter quantization:
- Keeps important weights precise
- Quantizes less important weights more
- Result: Better quality at same size!

The Economics:

Without Quantization:

• Llama-2-70B: Needs 8× A100 ($80K)
• Can't run on consumer hardware
• Limited to big companies

With Quantization:

• Llama-2-70B: Runs on RTX 4090 ($1.5K)
• Anyone can run it!
• Democratizes AI!

Impact: 50x cost reduction! 💰

Quantization Methods:

1. Post-Training Quantization (PTQ)

- Quantize after training
- Fast (minutes)
- Slight quality loss
- Most common

2. Quantization-Aware Training (QAT)

- Train with quantization in mind
- Slow (days)
- Minimal quality loss
- Best quality

3. Dynamic Quantization

- Quantize on-the-fly
- Flexible
- Good for deployment

By the end of this guide, you'll understand:

• How quantization works (bit precision reduction)
• Different quantization methods (GPTQ, GGUF, AWQ)
• How to quantize models yourself
• Quality vs. size trade-offs
• When to use INT8 vs. INT4 vs. INT3
• Real-world deployment strategies

The revolution: Run 70B models on your laptop! Quantization makes powerful AI accessible to everyone! ⚡

Let's dive in! 🚀

⚡ Model Quantization

Reduce model size and speed up inference by using lower-precision numbers

zation?](#what-is-quantization)

• Why Quantization Matters
• Core Concepts
• How It Works
• Quantization Methods
• Local Setup
• Interactive Tools & Visualizations
• Code Examples
• Hands-On Exercises
• Assignments & Practice
• Common Pitfalls & Debugging
• Fun Facts & Hacks
• Related Concepts
• Resources

What is Quantization?

Quantization is the process of reducing the precision of model weights and activations from high-precision formats (like FP32) to lower-precision formats (like INT8 or INT4). This dramatically reduces model size and speeds up inference with minimal accuracy loss.

Key Idea

FP32 (32-bit float)  →  INT8 (8-bit integer)
4 bytes per weight   →  1 byte per weight
= 4x smaller model, 2-4x faster inference!

Real-World Analogy

Think of it like image compression:

• Original: High-resolution RAW image (100MB)
• Quantized: Compressed JPEG (5MB)
• Result: 95% smaller, looks nearly identical
• Trade-off: Slight quality loss for huge size reduction

Why Quantization Matters

1. Massive Size Reduction 📦

• 4x smaller with INT8 (32-bit → 8-bit)
• 8x smaller with INT4 (32-bit → 4-bit)
• Deploy 70B models on consumer GPUs

2. Faster Inference ⚡

• 2-4x speedup with INT8
• Integer operations are faster than floating-point
• Better hardware utilization

3. Lower Memory Bandwidth 🚀

• Reduced data transfer
• Fit larger batches in memory
• Better cache utilization

4. Edge Deployment 📱

• Run models on mobile devices
• IoT and embedded systems
• Privacy-preserving local inference

5. Cost Savings 💰

• Cheaper hardware requirements
• Lower cloud costs
• Reduced energy consumption

Core Concepts

1. Number Representations

# FP32 (Float32): Standard precision
weight_fp32 = 0.123456789  # 32 bits, high precision

# FP16 (Float16): Half precision
weight_fp16 = 0.1235  # 16 bits, good precision

# INT8: 8-bit integer
weight_int8 = 31  # 8 bits, range: -128 to 127

# INT4: 4-bit integer
weight_int4 = 7  # 4 bits, range: -8 to 7

2. Quantization Formula

# Quantize: Float → Integer
quantized = round(float_value / scale) + zero_point

# Dequantize: Integer → Float
dequantized = (quantized - zero_point) * scale

3. Symmetric vs Asymmetric

Symmetric Quantization (zero_point = 0):

Range: [-127, 127]
scale = max(abs(weights)) / 127
quantized = round(weight / scale)

Asymmetric Quantization (zero_point ≠ 0):

Range: [-128, 127]
scale = (max - min) / 255
zero_point = round(-min / scale)
quantized = round(weight / scale) + zero_point

4. Per-Tensor vs Per-Channel

# Per-Tensor: Single scale for entire tensor
scale = tensor.abs().max() / 127

# Per-Channel: Different scale per output channel
scales = tensor.abs().max(dim=1) / 127  # Better accuracy

How It Works

Quantization Pipeline

graph TD
    A[FP32 Model] --> B[Calibration]
    B --> C[Compute Scales]
    C --> D[Quantize Weights]
    D --> E[INT8 Model]
    E --> F[Inference]
    F --> G[Dequantize if needed]
    G --> H[FP32 Output]

Mathematical Process

Step 1: Find Range

min_val = weights.min()
max_val = weights.max()

Step 2: Compute Scale

scale = (max_val - min_val) / (2^bits - 1)

Step 3: Quantize

quantized = round((weights - min_val) / scale)

Step 4: Store

# Store: quantized values + scale + zero_point
model_int8 = {
    'weights': quantized,
    'scale': scale,
    'zero_point': zero_point
}

Quantization Methods

1. Post-Training Quantization (PTQ)

Quantize after training, no retraining needed:

# Dynamic Quantization (weights only)
model_int8 = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear},
    dtype=torch.qint8
)

# Static Quantization (weights + activations)
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
model_prepared = torch.quantization.prepare(model)
# Run calibration
model_int8 = torch.quantization.convert(model_prepared)

2. Quantization-Aware Training (QAT)

Train with quantization in mind:

# Simulate quantization during training
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
model_prepared = torch.quantization.prepare_qat(model)

# Train normally
for epoch in range(num_epochs):
    train(model_prepared)

# Convert to quantized
model_int8 = torch.quantization.convert(model_prepared)

3. Mixed Precision

Different layers use different precisions:

# Sensitive layers: FP16
# Other layers: INT8
# Embeddings: INT4

quantization_config = {
    'attention': 'fp16',
    'ffn': 'int8',
    'embeddings': 'int4'
}

4. GPTQ (Generative Pre-trained Transformer Quantization)

Advanced quantization for LLMs:

from auto_gptq import AutoGPTQForCausalLM

# Quantize to 4-bit
model = AutoGPTQForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b",
    quantize_config={"bits": 4}
)

Local Setup

Installation (15-20 minutes)

# Create environment
conda create -n quantization python=3.10
conda activate quantization

# Install PyTorch with quantization support
pip install torch torchvision torchaudio

# Install quantization libraries
pip install bitsandbytes  # For 8-bit/4-bit quantization
pip install auto-gptq  # For GPTQ
pip install optimum  # Hugging Face optimization
pip install onnx onnxruntime  # For ONNX quantization

# Install transformers
pip install transformers accelerate

Quick Start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model in 8-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_8bit=True,
    device_map="auto"
)

# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_4bit=True,
    device_map="auto"
)

# Use normally
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
inputs = tokenizer("Hello, world!", return_tensors="pt")
outputs = model.generate(**inputs)

Code Examples

Example 1: Basic PyTorch Quantization (Beginner)

import torch
import torch.nn as nn

# Simple model
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 10)
        self.relu = nn.ReLU()
    
    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# Create and train model
model = SimpleModel()
# ... training code ...

# Dynamic Quantization (easiest)
model_int8 = torch.quantization.quantize_dynamic(
    model,
    {nn.Linear},  # Quantize Linear layers
    dtype=torch.qint8
)

# Compare sizes
def get_model_size(model):
    torch.save(model.state_dict(), "temp.p")
    size = os.path.getsize("temp.p") / 1e6
    os.remove("temp.p")
    return size

print(f"FP32 size: {get_model_size(model):.2f} MB")
print(f"INT8 size: {get_model_size(model_int8):.2f} MB")

# Use quantized model
output = model_int8(torch.randn(1, 784))

Example 2: Static Quantization with Calibration (Intermediate)

import torch
import torch.quantization as quantization

class ModelWithQuantStubs(nn.Module):
    """Model with quantization stubs"""
    def __init__(self):
        super().__init__()
        self.quant = quantization.QuantStub()
        self.fc1 = nn.Linear(784, 256)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(256, 10)
        self.dequant = quantization.DeQuantStub()
    
    def forward(self, x):
        x = self.quant(x)
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.dequant(x)
        return x

# Prepare model for quantization
model = ModelWithQuantStubs()
model.eval()

# Set quantization config
model.qconfig = quantization.get_default_qconfig('fbgemm')

# Prepare for calibration
model_prepared = quantization.prepare(model)

# Calibration: Run representative data through model
with torch.no_grad():
    for data, _ in calibration_dataloader:
        model_prepared(data)

# Convert to quantized model
model_int8 = quantization.convert(model_prepared)

# Save quantized model
torch.save(model_int8.state_dict(), 'model_int8.pth')

Example 3: Quantization-Aware Training (Advanced)

import torch
import torch.quantization as quantization

def train_with_qat(model, train_loader, num_epochs=10):
    """Quantization-Aware Training"""
    
    # Set QAT config
    model.qconfig = quantization.get_default_qat_qconfig('fbgemm')
    
    # Prepare for QAT
    model.train()
    model_prepared = quantization.prepare_qat(model)
    
    optimizer = torch.optim.Adam(model_prepared.parameters(), lr=0.001)
    criterion = nn.CrossEntropyLoss()
    
    # Training loop with fake quantization
    for epoch in range(num_epochs):
        for batch_idx, (data, target) in enumerate(train_loader):
            optimizer.zero_grad()
            output = model_prepared(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
            
            if batch_idx % 100 == 0:
                print(f'Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}')
        
        # Gradually freeze batch norm after a few epochs
        if epoch > 3:
            model_prepared.apply(torch.quantization.disable_observer)
        if epoch > 2:
            model_prepared.apply(torch.nn.intrinsic.qat.freeze_bn_stats)
    
    # Convert to quantized model
    model_prepared.eval()
    model_int8 = quantization.convert(model_prepared)
    
    return model_int8

# Usage
model = ModelWithQuantStubs()
model_int8 = train_with_qat(model, train_loader)

Example 4: 4-bit Quantization with bitsandbytes (Production)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",  # Normal Float 4
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True  # Nested quantization
)

# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Generate text
def generate_text(prompt, max_length=100):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_length=max_length,
            temperature=0.7,
            do_sample=True
        )
    
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Use the quantized model
result = generate_text("Once upon a time")
print(result)

# Check memory usage
print(f"Memory allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

Example 5: GPTQ Quantization for LLMs (Expert)

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import torch

# Quantization configuration
quantize_config = BaseQuantizeConfig(
    bits=4,  # 4-bit quantization
    group_size=128,  # Group size for quantization
    desc_act=False,  # Activation order
    damp_percent=0.01  # Damping factor
)

# Load model
model = AutoGPTQForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantize_config=quantize_config
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Prepare calibration data
def prepare_calibration_data(tokenizer, n_samples=128):
    """Prepare data for GPTQ calibration"""
    from datasets import load_dataset
    
    dataset = load_dataset("c4", "en", split="train")
    examples = []
    
    for i in range(n_samples):
        text = dataset[i]["text"]
        tokens = tokenizer(
            text,
            return_tensors="pt",
            max_length=2048,
            truncation=True
        )
        examples.append(tokens)
    
    return examples

# Calibrate and quantize
calibration_data = prepare_calibration_data(tokenizer)
model.quantize(calibration_data)

# Save quantized model
model.save_quantized("./llama-2-7b-gptq-4bit")
tokenizer.save_pretrained("./llama-2-7b-gptq-4bit")

# Load quantized model
model_quantized = AutoGPTQForCausalLM.from_quantized(
    "./llama-2-7b-gptq-4bit",
    device="cuda:0"
)

# Benchmark
import time

def benchmark(model, tokenizer, prompt, num_runs=10):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    # Warmup
    for _ in range(3):
        _ = model.generate(**inputs, max_new_tokens=50)
    
    # Benchmark
    start = time.time()
    for _ in range(num_runs):
        outputs = model.generate(**inputs, max_new_tokens=50)
    elapsed = time.time() - start
    
    tokens_per_second = (50 * num_runs) / elapsed
    print(f"Tokens/second: {tokens_per_second:.2f}")
    print(f"Time per generation: {elapsed/num_runs:.2f}s")

benchmark(model_quantized, tokenizer, "Hello, world!")

TODO: Compare different quantization levels

def analyze_quantization_impact(model, test_loader): """ Test model at different precisions: - FP32 (baseline) - FP16 - INT8 - INT4

Measure:
- Accuracy
- Model size
- Inference speed
- Memory usage
"""
# Your code here
pass

### Exercise 2: Custom Quantization
**Goal**: Implement quantization from scratch

```python
# TODO: Implement symmetric quantization
def symmetric_quantize(tensor, bits=8):
    """
    Quantize tensor using symmetric quantization
    
    Returns:
        quantized: INT tensor
        scale: Scaling factor
    """
    # Your code here
    pass

def symmetric_dequantize(quantized, scale):
    """Dequantize back to float"""
    # Your code here
    pass

Exercise 3: Mixed Precision Strategy

Goal: Design optimal mixed-precision configuration

# TODO: Find best precision for each layer
def find_optimal_precision(model, val_loader, accuracy_threshold=0.99):
    """
    Determine which layers can use lower precision
    while maintaining accuracy above threshold
    
    Returns:
        precision_map: Dict mapping layers to precisions
    """
    # Your code here
    pass

Exercise 4: Calibration Data Selection

Goal: Optimize calibration dataset

# TODO: Select best calibration samples
def select_calibration_data(full_dataset, n_samples=100):
    """
    Choose most representative samples for calibration
    
    Strategies:
    - Random sampling
    - Diverse sampling
    - Hard example mining
    """
    # Your code here
    pass

Exercise 5: Quantization Error Analysis

Goal: Understand where quantization hurts most

# TODO: Analyze per-layer quantization error
def analyze_quantization_errors(model_fp32, model_int8, test_data):
    """
    Compute and visualize:
    - Per-layer activation differences
    - Output distribution changes
    - Error accumulation through layers
    """
    # Your code here
    pass

Assignments & Practice

Assignment 1: Quantize BERT

Difficulty: ⭐⭐⭐☆☆

Quantize BERT for mobile deployment:

1. Load BERT-base model
2. Apply dynamic quantization
3. Apply static quantization with calibration
4. Compare accuracy and speed
5. Export to ONNX

Deliverables:

• Quantized models (INT8)
• Benchmark results
• ONNX model
• Deployment guide

Assignment 2: 4-bit LLM Deployment

Difficulty: ⭐⭐⭐⭐☆

Deploy Llama-2-7B in 4-bit:

1. Quantize using GPTQ
2. Optimize for inference
3. Build API endpoint
4. Benchmark throughput
5. Compare with FP16

Deliverables:

• Quantized model
• FastAPI server
• Load testing results
• Cost analysis

Assignment 3: Custom Quantization Research

Difficulty: ⭐⭐⭐⭐⭐

Implement and evaluate novel quantization:

1. Design custom quantization scheme
2. Implement in PyTorch
3. Test on multiple models
4. Compare with existing methods
5. Write research paper

Deliverables:

• Implementation code
• Experimental results
• Ablation studies
• Technical paper

Significant accuracy loss after quantization

fp32_accuracy = 0.95 int8_accuracy = 0.75 # Too low!

**Solutions**:
```python
# 1. Use QAT instead of PTQ
model_int8 = train_with_qat(model, train_loader)

# 2. Better calibration data
calibration_data = select_diverse_samples(dataset, n=1000)

# 3. Per-channel quantization
use_per_channel_quantization()

# 4. Mixed precision for sensitive layers
keep_attention_in_fp16()

Issue 2: Slow Quantized Inference

Symptoms:

# Quantized model not faster
fp32_time = 100ms
int8_time = 95ms  # Expected 2-4x speedup!

Solutions:

# 1. Use correct backend
torch.backends.quantized.engine = 'fbgemm'  # x86
# or
torch.backends.quantized.engine = 'qnnpack'  # ARM

# 2. Fuse operations
model = torch.quantization.fuse_modules(model, [['conv', 'bn', 'relu']])

# 3. Use static quantization (not dynamic)
model_int8 = static_quantize(model, calibration_data)

# 4. Batch inputs
outputs = model(batched_inputs)  # Faster than loop

Issue 3: Memory Issues

Symptoms:

# OOM during quantization
RuntimeError: CUDA out of memory

Solutions:

# 1. Quantize on CPU first
model = model.cpu()
model_int8 = quantize(model)
model_int8 = model_int8.cuda()

# 2. Quantize layer by layer
for name, module in model.named_modules():
    if isinstance(module, nn.Linear):
        quantize_module(module)

# 3. Use gradient checkpointing
model.gradient_checkpointing_enable()

# 4. Smaller calibration batch
calibration_batch_size = 8  # Reduce from 32

Issue 4: Incompatible Operations

Symptoms:

# Some operations don't support INT8
RuntimeError: Operation not supported for quantized tensors

Solutions:

# 1. Add dequant/quant stubs around unsupported ops
class QuantizedModel(nn.Module):
    def forward(self, x):
        x = self.quant(x)
        x = self.supported_op(x)
        x = self.dequant(x)
        x = self.unsupported_op(x)  # Runs in FP32
        x = self.quant(x)
        return x

# 2. Use mixed precision
quantize_config = {
    'supported_layers': 'int8',
    'unsupported_layers': 'fp32'
}

# 3. Replace with quantized-friendly operations
# Replace: LayerNorm → GroupNorm
# Replace: GELU → ReLU

Hack 1: Smooth quantization

Reduce outliers before quantization

def smooth_weights(model, alpha=0.5): for module in model.modules(): if hasattr(module, 'weight'): w = module.weight.data w_smooth = w * alpha + w.mean() * (1 - alpha) module.weight.data = w_smooth

Hack 2: Outlier-aware quantization

Handle outliers separately

def quantize_with_outliers(tensor, threshold=6.0): outliers = tensor.abs() > threshold normal = ~outliers

# Quantize normal values
tensor[normal] = quantize(tensor[normal])
# Keep outliers in FP16
tensor[outliers] = tensor[outliers].half()

Hack 3: Dynamic range adjustment

Adjust quantization range during inference

class AdaptiveQuantization: def init(self): self.running_min = None self.running_max = None

def update_range(self, tensor, momentum=0.9):
    if self.running_min is None:
        self.running_min = tensor.min()
        self.running_max = tensor.max()
    else:
        self.running_min = momentum * self.running_min + (1-momentum) * tensor.min()
        self.running_max = momentum * self.running_max + (1-momentum) * tensor.max()

Hack 4: Quantization-friendly architecture

Design models that quantize well

class QuantFriendlyBlock(nn.Module): def init(self): super().init() # Use ReLU instead of GELU (better for quantization) self.activation = nn.ReLU() # Use GroupNorm instead of LayerNorm self.norm = nn.GroupNorm(32, 512) # Avoid operations with large dynamic range self.linear = nn.Linear(512, 512)

Hack 5: Post-quantization fine-tuning

Quick recovery of accuracy

def post_quant_finetune(model_int8, train_loader, num_steps=1000): """Fine-tune for just 1000 steps to recover accuracy""" optimizer = torch.optim.Adam(model_int8.parameters(), lr=1e-5)

for step, batch in enumerate(train_loader):
    if step >= num_steps:
        break
    loss = compute_loss(model_int8, batch)
    loss.backward()
    optimizer.step()

### 🔧 Debugging Tricks

```python
# Visualize quantization error per layer
def debug_quantization(model_fp32, model_int8, sample_input):
    """Compare layer-by-layer outputs"""
    
    fp32_activations = {}
    int8_activations = {}
    
    # Hook to capture activations
    def get_activation(name, storage):
        def hook(model, input, output):
            storage[name] = output.detach()
        return hook
    
    # Register hooks
    for name, module in model_fp32.named_modules():
        module.register_forward_hook(get_activation(name, fp32_activations))
    
    for name, module in model_int8.named_modules():
        module.register_forward_hook(get_activation(name, int8_activations))
    
    # Forward pass
    _ = model_fp32(sample_input)
    _ = model_int8(sample_input)
    
    # Compare
    for name in fp32_activations:
        if name in int8_activations:
            fp32_act = fp32_activations[name]
            int8_act = int8_activations[name]
            
            error = (fp32_act - int8_act).abs().mean()
            print(f"{name}: Error = {error:.6f}")

Progress Tracking

• Understand quantization fundamentals
• Implement basic INT8 quantization
• Apply PTQ to a model
• Try QAT for better accuracy
• Quantize an LLM to 4-bit
• Benchmark quantized models
• Deploy quantized model to production
• Explore advanced techniques (GPTQ, mixed precision)