How does a token dynamically gather context from other words in a sentence? Type any sentence below, select a token to be the Query, and observe the live attention connections, softmax weights, and $N \times N$ attention matrix.
Select any token above. Arcs illustrate attention flow to all sequence tokens; brighter and thicker arcs indicate higher attention weights.
Each row represents the query token attending across all key columns. Each row sums to ~100%.
What is Attention Simulator? Why does it exist? Where does it appear in Attention?
Draw or describe a visual model of Attention Simulator. What does it look like conceptually?
Work through a concrete manual example of Attention Simulator with real numbers or pseudocode.
Implement Attention Simulator from scratch in Python. Keep it minimal — just enough to see it work.
Try Attention Simulator on a real example. Vary the inputs and observe what changes.
How is Attention Simulator different from related concepts in Attention? What are the trade-offs?
Explain Attention Simulator in plain language using an analogy. Could a non-expert follow it?
Build a small project or demo that uses Attention Simulator as a core component.
agent/state_tracker.py
📋 full plan →