Type any sentence and watch it split into subword tokens using greedy longest-match BPE — just like a real tokenizer pipeline.
Try this experiment → ① Type a word that's not in the vocabulary — e.g. supercalifragilistic ② Hit Tokenize — watch it get broken into single characters (unknown fallback) ③ Scroll to 📚 Vocabulary Table → type your word → click + Add ④ Tokenize again — it now maps to a single token with one ID ✨
⚠️ Unknown tokens detected (shown with ? ID) — the tokenizer fell back to single characters.
Scroll to 📚 Vocabulary Table below, add the word, then tokenize again to see it resolve into a single token.
🎨 Token Visualization
tokens will appear here…
0
Tokens
0
Words
0
Chars
—
Tok/Word
🔢 Token ID Array
[]
📚 Vocabulary Table
💡 Add your unknown word here — type it in the field below, optionally give it an ID, then click + Add.
Head back to the input and hit Tokenize again to see it map to a single token instead of character fragments.
