🔨 BUILD: Your Own Tokenizer

🎯 What You'll Build

1. ✅ BPE tokenizer from scratch
2. ✅ Token cost calculator
3. ✅ Tokenizer comparison tool

📋 Setup

pip install transformers tiktoken

🚀 Project 1: Token Cost Calculator

import tiktoken

class TokenCostCalculator:
    def __init__(self, model="gpt-4"):
        self.encoding = tiktoken.encoding_for_model(model)
        self.prices = {
            "gpt-4": {"input": 30, "output": 60},
            "gpt-3.5-turbo": {"input": 0.5, "output": 1.5},
        }
        self.model = model
    
    def count_tokens(self, text):
        return len(self.encoding.encode(text))
    
    def calculate_cost(self, input_text, output_text):
        input_tokens = self.count_tokens(input_text)
        output_tokens = self.count_tokens(output_text)
        
        input_cost = (input_tokens / 1_000_000) * self.prices[self.model]["input"]
        output_cost = (output_tokens / 1_000_000) * self.prices[self.model]["output"]
        
        return {
            "input_tokens": input_tokens,
            "output_tokens": output_tokens,
            "total_cost": input_cost + output_cost
        }

# Use it
calc = TokenCostCalculator("gpt-4")
cost = calc.calculate_cost("Hello", "Hi there!")
print(f"Cost: ${cost['total_cost']:.6f}")

🚀 Project 2: Tokenizer Comparison

from transformers import AutoTokenizer

class TokenizerComparison:
    def __init__(self):
        self.tokenizers = {
            "GPT-2": AutoTokenizer.from_pretrained("gpt2"),
            "BERT": AutoTokenizer.from_pretrained("bert-base-uncased"),
        }
    
    def compare(self, text):
        for name, tokenizer in self.tokenizers.items():
            tokens = tokenizer.tokenize(text)
            print(f"{name}: {len(tokens)} tokens")
            print(f"  {tokens}\n")

# Use it
comp = TokenizerComparison()
comp.compare("Hello, how are you?")

🎯 Challenges

Challenge 1: Optimize Prompts

Rewrite prompts to use fewer tokens and save money.

Challenge 2: Build Simple BPE

Implement basic BPE merge operation.

Challenge 3: Multilingual Support

Test tokenizers with Chinese, Arabic, Hindi text.

🔗 Related

• 01.a BPE - BPE algorithm details
• Vectorization - Next step

Congratulations! You've completed the Tokenization module. Next: Vectorization