//
tokenization
STEP 01
What is a token?
⏸ Pause
⏭ Step
↺ Reset
Speed
0.5×
🌙
01 · What is a token?
›
02 · Text splitting
›
03 · BPE merges
›
04 · Vocabulary lookup
›
05 · Embed → numbers
›
06 · Meaning space
›
07 · Special tokens
›
08 · Full pipeline
STEP 01 · TOKENIZE
// STEP 01 · WHAT IS A TOKEN?
A token is the smallest unit an LLM can read.
Not words. Not characters. Something in between.
This page
0:00
This session
0:00
Total (all pages)
0:00
Today
0:00
▶ Running
📊 View all study stats →
0:00
studying