Interactive node mindmap meets deep architectural walkthrough. Explore the full progression from raw prompt ingestion through BPE tokenization, embedding geometry, multi-head attention, autoregressive prediction, and formatted delivery.
Every interaction begins as unstructured data. The application layer compiles the system instructions, conversation history, retrieved knowledge (RAG), and multimodal attachments (images, audio, files) into a unified context packet. Input validation and prompt injection defenses sanitize this payload before handing off to the tokenizer.
Computers cannot process raw characters directly. The tokenizer applies Byte Pair Encoding (BPE) or WordPiece to split words into frequent subword units, avoiding out-of-vocabulary errors while preserving morphologic stems. Each token is assigned a unique integer ID from a fixed vocabulary (e.g. 32,000 to 128,000 unique tokens).
Each integer ID acts as a row index into a massive learned weight matrix $W_E$. The lookup converts a discrete ID into a rich vector of floating-point numbers (e.g. 768 to 4,096 dimensions). Tokens with similar grammatical roles or semantic meanings are placed close together in this high-dimensional vector space.
Self-attention has no innate sense of sequence: "Charan planted a thought in Advaitha" looks identical to "Advaitha planted a thought in Charan". Positional information solves this by either adding sinusoidal/learned position vectors or multiplying by 2D rotational transformation matrices (RoPE in LLaMA / Mistral / DeepSeek).
Modern LLMs (like GPT-4, Claude, LLaMA) are decoder-only autoregressive Transformers. They stack 32 to 128 identical Transformer blocks. The residual stream acts as an information superhighway that allows gradients to flow directly through hundreds of layers without vanishing.
For every token, the network projects three vectors: Query ($Q$) (what am I looking for?), Key ($K$) (what do I offer?), and Value ($V$) (what information do I contain?). The dot product of $Q$ and $K$ measures similarity. After scaling and applying a causal mask $M$, a softmax produces attention weights that blend the $V$ vectors into a contextualized state.
The model synthesizes the entire sequence representation into an integrated semantic state. It resolves coreferences (e.g. "head" belongs to "Advaitha"), follows user directives, disambiguates complex constraints, and retains facts across lengthy multi-turn conversations.
The final hidden vector $\mathbf{h}_{final}$ at the last sequence position is multiplied by the unembedding matrix $W_U$ to generate raw scores (logits) for all tokens in the vocabulary. Softmax normalizes them into a probability distribution. Decoding strategies (Greedy, Temperature, Top-K, Top-P Nucleus) select the single next winner.
Generation is autoregressive: the model generates one token at a time. The newly predicted token is appended to the input sequence, and the full forward pass repeats. KV caching stores prior key-value attention vectors to avoid redundant computation, speeding up inference by orders of magnitude.
The reverse of tokenization. As token IDs emerge from the sampling loop, the detokenizer maps each integer back to its corresponding character or byte chunk. Subword fragments are stitched together, unicode byte boundaries are aligned, leading whitespace flags (e.g. Ġ or ) are turned into spaces, and control tokens are stripped.
The final reconstructed string passes through the application safety and display layers. Output guardrails check for toxic content or hallucinations. If the model generated a structured tool call (e.g. running code or calling a database), the agent system intercepts it, executes the action, and loops the result back into Stage 1.