🎲 LLM Text Generation

Real-world scenario: You type "The capital of France is" into ChatGPT. The model doesn't look up an answer — it produces 50,000 numbers (one per vocabulary token), converts them into probabilities, and picks the next word. That word gets appended, and the whole process repeats. Every word, comma, and hallucination you've ever seen is this loop running hundreds of times. That loop is text generation.

🌟 Start With a Real-World Scenario

The generation loop — one cycle:

Input: "The capital of France is"
       ↓
Transformer forward pass
       ↓
Logits: [−1.2, 8.4, 3.1, 0.5, 1.0, ...]   ← 50,000 raw scores
       ↓
Softmax → Probabilities: [0.0%, 99.5%, 0.4%, ...]
       ↓
Pick a token  →  "Paris"
       ↓
New input: "The capital of France is Paris"
       ↓  (repeat until stop condition)

The question is: how do you pick that token? That's what this doc is about.

Logits

When the transformer finishes processing your input, the very last layer produces a vector of raw numbers — one number per token in the vocabulary. These are called logits.

A logit of 8.4 for "Paris" and 3.1 for "Rome" means the model strongly prefers "Paris" — but these aren't probabilities yet. They can be negative, large, or small. They don't sum to 1.

import torch

vocab  = [".", "Paris", "Rome", "a", "the"]
logits = torch.tensor([-1.2, 8.4, 3.1, 0.5, 1.0])

top_idx = logits.argmax().item()
print(f"Highest logit: '{vocab[top_idx]}'")  # → Paris

Probabilities

To convert logits into proper probabilities, we apply softmax: exponentiate every value and divide by the sum. Result sums to 1.

import torch.nn.functional as F

probs = F.softmax(logits, dim=-1)
for t, p in zip(vocab, probs):
    print(f"  {t:6s}  {p.item():.4f}")
# .       0.0000
# Paris   0.9949  ← dominates
# Rome    0.0041
# a       0.0001
# the     0.0002

The exponential amplifies differences dramatically. A logit gap of 5.3 → probability ratio of ~200×. That's why one token can absorb 99 % of probability mass.

Greedy Decoding

The simplest strategy: always pick the highest-probability token. Deterministic, fast, reproducible.

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
model.eval()

input_ids = tokenizer.encode("The capital of France is", return_tensors="pt")
with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=5, do_sample=False)
print(tokenizer.decode(output[0]))
# → "The capital of France is Paris." (same every time)

Downside: commits to the best token at every step — never explores alternatives. Can produce repetitive or dull text.

Sampling

Instead of always picking the top token, treat the probabilities as a lottery and draw randomly. A token with 90 % probability wins 90 % of the time.

output = model.generate(input_ids, max_new_tokens=15, do_sample=True)
print(tokenizer.decode(output[0]))
# Different every run — sometimes creative, sometimes weird

Pure sampling can occasionally pick bizarre low-probability tokens. Temperature, Top-K, and Top-P tame this.

Temperature

Temperature scales the logits before softmax: scaled_logits = logits / T

• T < 1 (e.g. 0.3): sharpens distribution → model is more conservative, predictable
• T = 1: no change
• T > 1 (e.g. 1.5): flattens distribution → model is more creative, more prone to errors

for T in [0.3, 1.0, 2.0]:
    p = F.softmax(logits / T, dim=-1)
    print(f"T={T}: Paris={p[1]:.3f}, Rome={p[2]:.3f}")
# T=0.3: Paris=1.000, Rome=0.000
# T=1.0: Paris=0.995, Rome=0.004
# T=2.0: Paris=0.797, Rome=0.138

Most production systems use T = 0.7–1.0. T < 0.5 starts to feel robotic; T > 1.2 starts to feel random.

Top-K Sampling

Keep only the K most probable tokens before sampling. Everything else gets probability 0.

def top_k_sample(logits, k):
    top_k_vals, top_k_idx = torch.topk(logits, k)
    filtered = torch.full_like(logits, float("-inf"))
    filtered.scatter_(0, top_k_idx, top_k_vals)
    probs = F.softmax(filtered, dim=-1)
    return torch.multinomial(probs, 1).item()

idx = top_k_sample(logits, k=2)  # only "Paris" or "Rome" can be sampled
print(vocab[idx])

Weakness: K is fixed regardless of context. Sometimes 2 tokens are right, sometimes 50. Top-P solves this.

Top-P (Nucleus) Sampling

Instead of fixing the count, fix the cumulative probability. Sort tokens by probability, keep adding until their cumulative sum reaches P. Sample only from that "nucleus".

• High-confidence context → nucleus shrinks (maybe 2-3 tokens)
• Low-confidence context → nucleus grows (maybe 50+ tokens)
• Adapts automatically to the situation

def top_p_sample(logits, p):
    probs = F.softmax(logits, dim=-1)
    sorted_probs, sorted_idx = torch.sort(probs, descending=True)
    cumprobs = torch.cumsum(sorted_probs, dim=0)
    remove = cumprobs - sorted_probs > p
    sorted_probs[remove] = 0.0
    sorted_probs /= sorted_probs.sum()
    sampled = torch.multinomial(sorted_probs, 1).item()
    return sorted_idx[sampled].item()

idx = top_p_sample(logits, p=0.9995)
print(vocab[idx])

Most LLM APIs default to top_p=0.9 or top_p=1.0. Often combined with temperature.

Stop Conditions

An LLM doesn't know when to stop — left running, it fills the entire context window. Stop conditions halt the loop.

output = model.generate(
    input_ids,
    max_new_tokens=50,          # hard cap
    do_sample=True,
    temperature=0.9,
    top_p=0.95,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

A missing stop condition is one of the most common deployment bugs — the model loops or fills the context with repetition.