🎴 Kickstart FlashCards

Spaced Repetition Learning

These flashcards help you memorize key concepts from the AI Kickstart section. Use Obsidian's Spaced Repetition plugin to review these cards regularly.

📚 AI Winter to Summer

Card 1: AI Winter Definition

What is the "AI Winter"? ? A period (1970s-1980s and 1990s-2000s) when AI research funding dried up due to unmet expectations and limited computing power. Progress stalled because:

Computers were too slow
Data was scarce
Algorithms were basic

Card 2: Three Breakthroughs

What are the THREE key breakthroughs that ended the AI Winter? ?

Computing Power - GPUs made training 100x faster
Big Data - Internet provided billions of text examples
Transformer Architecture (2017) - Attention mechanism that processes all words simultaneously

Card 3: GPU Impact

Why were GPUs crucial for AI's breakthrough? ? GPUs (Graphics Processing Units) can perform thousands of calculations in parallel, making neural network training 100x faster than CPUs. Originally designed for gaming graphics, they became perfect for matrix operations in deep learning.

Card 4: Attention Mechanism

What problem does the Attention mechanism solve? ? Traditional RNNs/LSTMs process words sequentially (one at a time), losing context in long sentences. Attention allows the model to look at ALL words simultaneously and decide which words are most important for understanding context.

Card 5: BERT vs GPT

What's the key difference between BERT and GPT? ?

BERT (2018): Bidirectional - reads text in both directions, best for understanding (classification, Q&A)
GPT (2018): Unidirectional - reads left-to-right, best for generation (writing, completion)

🖥️ Traditional vs AI Servers

Card 6: Rule-Based Server

How does a traditional rule-based server work? ? Uses explicit IF-THEN rules written by programmers:

IF question contains "weather":
    RETURN weather_data
ELSE IF question contains "time":
    RETURN current_time

Limited to predefined scenarios, breaks with unexpected input.

Card 7: AI Server Advantage

What's the main advantage of an AI-powered server over rule-based? ? AI servers learn patterns from data instead of following hardcoded rules. They can:

Handle unexpected inputs
Understand context and nuance
Improve with more data
Generalize to new situations

Card 8: BERT Server Example

What can a BERT-powered server do that a rule-based server cannot? ? BERT understands semantic meaning, not just keywords:

"What's the climate like?" → Understands this asks about weather
"How's it looking outside?" → Recognizes weather question
"Is it jacket weather?" → Infers temperature question Rule-based servers would miss these variations.

🔤 Language Models (LM vs LLM)

Card 9: Language Model Definition

What is a Language Model? ? A model that predicts the next word in a sequence based on previous words. It learns probability distributions: P(word | previous_words) Example: "The cat sat on the ___" → predicts "mat" (high probability)

Card 10: N-gram Model

How does an N-gram language model work? ? Counts word sequences in training data to calculate probabilities:

Bigram: P(word | previous_1_word)
Trigram: P(word | previous_2_words)

Example: "I love" → counts how often "pizza", "coding", etc. follow Limitation: Only looks at last N-1 words, no long-term memory

Card 11: LSTM Advantage

What advantage do LSTMs have over N-grams? ? LSTMs have memory cells that can remember information from much earlier in the sequence:

N-gram: Only sees last 2-3 words
LSTM: Can remember context from 100+ words ago
Uses gates (forget, input, output) to control what to remember/forget

Card 12: LLM Definition

What makes a model a "Large Language Model" (LLM)? ? An LLM is a language model with:

Transformer architecture (attention mechanism)
Billions of parameters (GPT-3: 175B, GPT-4: >1T)
Trained on massive datasets (entire internet)
Emergent abilities (reasoning, coding, translation) not explicitly trained

Card 13: Transformer Key Innovation

What's the key innovation of the Transformer architecture? ? Self-Attention mechanism: Allows the model to weigh the importance of every word relative to every other word in the sentence simultaneously (parallel processing).

Example: "The animal didn't cross the street because it was too tired" Attention helps identify "it" refers to "animal", not "street"

🎯 Key Concepts

Card 14: Pre-training

What is pre-training in the context of LLMs? ? Training a model on a massive general dataset (like all of Wikipedia, books, internet) to learn language patterns, facts, and reasoning. This creates a foundation model that can be fine-tuned for specific tasks.

Card 15: Fine-tuning

What is fine-tuning? ? Taking a pre-trained model and training it further on a specific task or domain with a smaller, specialized dataset. Much faster and cheaper than training from scratch.

Example: Pre-trained GPT → Fine-tune on medical texts → Medical AI assistant

Card 16: Tokenization

What is tokenization and why is it important? ? Breaking text into smaller units (tokens) that the model can process:

Word-level: "Hello world" → ["Hello", "world"]
Subword (BPE): "unhappiness" → ["un", "happiness"]

Important because:

Models work with numbers, not text
Subword tokenization handles rare words better
Reduces vocabulary size

Card 17: Embedding

What is an embedding? ? A dense vector representation of a token/word that captures its meaning:

"king" → [0.2, 0.8, -0.3, ...] (hundreds of dimensions)
Similar words have similar vectors
Enables mathematical operations: king - man + woman ≈ queen

Learned during training to represent semantic relationships.

Card 18: Context Window

What is a context window? ? The maximum number of tokens a model can process at once:

GPT-3: 4,096 tokens (~3,000 words)
GPT-4: 8,192 - 32,768 tokens
Claude 2: 100,000 tokens

Longer context = better understanding of long documents, but slower and more expensive.

🔧 Training Concepts

Card 19: Backpropagation

What is backpropagation? ? The algorithm used to train neural networks:

Forward pass: Input → predictions
Calculate loss (error)
Backward pass: Propagate error back through layers
Update weights to reduce error

Repeats millions of times until model learns patterns.

Card 20: Gradient Descent

What is gradient descent? ? An optimization algorithm that adjusts model weights to minimize loss:

Calculates gradient (direction of steepest increase in loss)
Moves weights in opposite direction (downhill)
Learning rate controls step size

Variants: SGD (stochastic), Mini-batch, Adam (adaptive)

Card 21: Overfitting

What is overfitting and how do you prevent it? ? When a model memorizes training data instead of learning patterns, performing poorly on new data.

Prevention:

Dropout: Randomly disable neurons during training
Regularization: Penalize large weights
More data: Harder to memorize
Early stopping: Stop before overfitting occurs

📊 Evaluation Metrics

Card 22: Perplexity

What is perplexity in language models? ? A measure of how "surprised" the model is by the test data:

Lower perplexity = better model
Perplexity of 10 = model is as confused as if choosing from 10 equally likely words
Formula: 2^(average_negative_log_likelihood)

Good models: perplexity 20-40 on common text

Card 23: Temperature Sampling

What does temperature control in text generation? ? Temperature controls randomness in generation:

Temperature = 0: Always pick most likely word (deterministic, boring)
Temperature = 0.7: Balanced creativity and coherence
Temperature = 1.0: Sample from full probability distribution
Temperature > 1.0: More random, creative, but less coherent

Used in generation, not training.

🚀 Modern AI Timeline

Card 24: Key Milestones

What are the key AI milestones from 2017-2023? ?

2017: Transformer architecture ("Attention is All You Need")
2018: BERT (bidirectional) and GPT-1 (generative)
2019: GPT-2 (1.5B parameters)
2020: GPT-3 (175B parameters) - few-shot learning
2022: ChatGPT - conversational AI goes mainstream
2023: GPT-4, Claude 2, Llama 2 - multimodal, longer context

Card 25: Emergent Abilities

What are emergent abilities in LLMs? ? Capabilities that appear in large models but not in smaller ones, without explicit training:

Chain-of-thought reasoning: Breaking down complex problems
Few-shot learning: Learning from just a few examples
Code generation: Writing functional code
Translation: Between languages never seen together

Emerge around 10B+ parameters.

💡 Quick Facts

Card 26: Parameter Count

What are parameters in a neural network? ? Weights and biases that the model learns during training:

Each connection between neurons has a weight
GPT-3: 175 billion parameters
GPT-4: Estimated 1+ trillion parameters
More parameters = more capacity to learn patterns (but needs more data/compute)

Card 27: Training Cost

Why is training LLMs expensive? ? Requires massive computational resources:

GPT-3 training: ~$4-5 million, 355 GPU-years
Electricity: Thousands of GPUs running for months
Data: Petabytes of text data
Expertise: ML engineers, researchers

Fine-tuning is much cheaper (thousands vs millions).

Card 28: Inference vs Training

What's the difference between training and inference? ?

Training: Learning from data, adjusting weights (expensive, one-time)
Inference: Using trained model to make predictions (cheap, repeated)

Example: Training GPT-3 cost millions, but each ChatGPT response costs ~$0.002

🎓 Study Tips

How to use these flashcards:

Install the Obsidian Spaced Repetition plugin
Review cards daily (takes 5-10 minutes)
The plugin will schedule reviews based on your performance
Cards you struggle with appear more frequently
Cards you master appear less often

Active recall is key! Try to answer before revealing the answer.

📝 Related Notes

00 The AI Winter to Summer
01 Why AI Now
02 Traditional vs AI Servers
09 LM vs LLM Deep Dive
01.a BUILD Simple Language Model
01.b BUILD Mini LLM