Tokens — My Understanding
Everything on this page is written by me, as I understand it. Not copied from the lessons — just what I took away after understanding how tokenization turns raw text into discrete integers for neural network lookup.
Big picture: What are tokens, in one go? #charan
The input text given to an AI is first converted into tokens (subword text chunks), and then the tokenizer looks up each token in its vocabulary table to get an integer Token ID.
The Token ID is then used directly as a row index into the embedding table matrix ($W_E$) to pull out its initial feature vector (see my embedding notes).
The tokenizer does not know the meaning of words. "plant" in "I plant an idea into Rahul's mind" and "plant" in "The green plant needs water" receives the exact same Token ID (e.g. 48365). The meaning is sorted out later in the Self-Attention stage (see my attention notes).
A token is the fundamental atomic unit of text that a Large Language Model reads and generates. It can be a whole word ("head"), a subword piece ("plant" + "ed"), a syllable ("Adva" + "itha"), or a punctuation mark (".").
The Three Options:
- Character-level (26 letters): Too granular. Sentences take thousands of tokens, self-attention matrices explode in compute ($O(N^2)$), and individual characters have no semantic meaning.
- Word-level (Full dictionary words): Vocabulary would need to be infinite to cover every plural, verb tense, typo, and rare name. Any unknown word results in an
[UNK](Out of Vocabulary) failure. - Subword-level / BPE (The Sweet Spot): Common words stay whole; rare or complex words break down into familiar subwords. Nothing is ever Out of Vocabulary!
Computers cannot do matrix multiplication on letters or strings. Every unique token in the tokenizer's dictionary has an assigned integer index called a Token ID.
In our unified example sentence: "Charan planted a new thought in Advaitha's head."
"Charan"→7490"␣planted"→12785(notice the leading space marker␣)"␣a"→257"␣new"→649"␣thought"→1807"␣in"→287"␣Adva"→18451+"itha"→893"'s"→338"␣head"→1182"."→13
The tokenizer outputs a 1D tensor: [7490, 12785, 257, 649, 1807, 287, 18451, 893, 338, 1182, 13].
Byte Pair Encoding builds the vocabulary bottom-up during pre-training by iteratively merging the most frequent adjacent byte or character pairs:
- Start with a base vocabulary of individual characters/bytes.
- Count all adjacent pairs across massive text datasets.
- Merge the most frequent pair (e.g.
t + h → th,t + h + e → the) and add it to the vocabulary. - Repeat thousands of times until reaching target vocabulary size (e.g. 50,257 tokens).