How Tokenization Works Beginner

Tokenization is the process of converting raw text into a sequence of tokens that an AI model can process. Several algorithms exist, each with different approaches to breaking text into subword units.

Tokenization Algorithms

There are four main tokenization algorithms used by modern LLMs:

🔁

BPE (Byte Pair Encoding)

Used by OpenAI (GPT models) and Meta (LLaMA). Starts with individual bytes and iteratively merges the most frequent pairs.

📜

WordPiece

Used by Google (BERT, early models). Similar to BPE but uses likelihood-based merging instead of frequency.

🌐

SentencePiece

Used by Google (Gemini, T5) and Meta (LLaMA). Language-agnostic; treats input as raw bytes, no pre-tokenization needed.

📊

Unigram

Used alongside SentencePiece. Starts with a large vocabulary and prunes tokens by removing those that least affect the loss.

How BPE Training Works

Byte Pair Encoding is the most widely used algorithm, so let us walk through how it works step by step:

Start with individual characters (or bytes)
The initial vocabulary contains every unique character in the training corpus. For byte-level BPE, it starts with all 256 possible bytes.
Count all adjacent pairs
Scan the entire corpus and count how often every pair of adjacent tokens appears together.
Merge the most frequent pair
The pair that appears most often is merged into a single new token and added to the vocabulary.
Repeat until vocabulary size is reached
Steps 2-3 repeat thousands of times until the desired vocabulary size (e.g., 100,000 tokens) is reached.

BPE Example

# Training corpus: "low lower lowest"
# Step 0: Start with characters
Vocabulary: [l, o, w, e, r, s, t, ' ']

# Step 1: Most frequent pair is ('l', 'o') → merge into 'lo'
Vocabulary: [l, o, w, e, r, s, t, ' ', lo]

# Step 2: Most frequent pair is ('lo', 'w') → merge into 'low'
Vocabulary: [l, o, w, e, r, s, t, ' ', lo, low]

# Step 3: Most frequent pair is ('e', 'r') → merge into 'er'
Vocabulary: [l, o, w, e, r, s, t, ' ', lo, low, er]

# Step 4: Most frequent pair is ('low', 'er') → merge into 'lower'
Vocabulary: [l, o, w, e, r, s, t, ' ', lo, low, er, lower]

# ... continues until target vocabulary size

Why BPE Works Well: BPE naturally learns that common words should be single tokens (efficient) while rare words can be broken into known subword pieces (flexible). This gives it both efficiency and the ability to handle any input text.

Vocabulary Size

The vocabulary size is a key design decision. Larger vocabularies mean more common words are single tokens (fewer tokens per text), but the model's embedding layer is bigger (more parameters).

Vocabulary Size	Used By	Trade-offs
32,000	LLaMA 2, early models	Smaller embedding, more tokens per text, weaker multilingual
50,257	GPT-2	Good balance for English-centric models
100,256	GPT-4 (cl100k_base)	Strong multilingual, efficient for code
128,000	LLaMA 3	Improved multilingual, more efficient tokenization
200,019	GPT-4o (o200k_base)	Best multilingual support, most efficient per text

Special Tokens

Beyond regular text tokens, every tokenizer includes special tokens that serve control functions:

Common Special Tokens

# BERT-style special tokens
[CLS]    → Start of sequence (classification token)
[SEP]    → Separator between segments
[PAD]    → Padding for batch alignment
[MASK]   → Masked token for pre-training
[UNK]    → Unknown token (out of vocabulary)

# GPT-style special tokens
<|endoftext|>      → End of document
<|im_start|>       → Start of message (chat format)
<|im_end|>         → End of message (chat format)
<|system|>         → System message marker

# Claude-style special tokens
\n\nHuman:          → User turn marker
\n\nAssistant:      → Assistant turn marker

Important: Special tokens are not generated by the tokenizer from regular text. They are explicitly inserted by the API or framework. You generally do not need to worry about them, but they do consume part of your context window.

Subword Splitting in Practice

Understanding how words get split helps you write more token-efficient prompts:

Word	Token Split	Why
`the`	`["the"]` - 1 token	Extremely common, learned as single token
`running`	`["running"]` - 1 token	Common enough to be a single token
`tokenization`	`["token", "ization"]` - 2 tokens	Split at morpheme boundary
`cryptocurrency`	`["crypt", "ocurrency"]` - 2 tokens	Moderately common compound word
`pneumonoultramicroscopicsilicovolcanoconiosis`	`["pne", "um", "ono", "ultra", ...]` - 11+ tokens	Very rare, splits into many pieces

Exploring Tokenization with tiktoken

Python

import tiktoken

# Load the tokenizer used by GPT-4
enc = tiktoken.get_encoding("cl100k_base")

# Encode text to tokens
text = "Tokenization is fascinating!"
tokens = enc.encode(text)
print(f"Token IDs: {tokens}")
# Output: Token IDs: [3404, 2065, 374, 27387, 0]

# Decode individual tokens to see the text pieces
for token in tokens:
    print(f"  {token} → '{enc.decode([token])}'")
# Output:
#   3404  → 'Token'
#   2065  → 'ization'
#   374   → ' is'
#   27387 → ' fascinating'
#   0     → '!'

# Count tokens
print(f"Total tokens: {len(tokens)}")
# Output: Total tokens: 5

Install tiktoken: Run pip install tiktoken to install the library. It is fast and works offline - the tokenizer runs locally, no API calls needed.

← Introduction Tokenizers by Model →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka

📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp

🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX