Introduction to Tokens Beginner

Tokens are the fundamental unit of text that AI language models work with. They are not words, not characters, but subword units - and understanding them is essential for working effectively with any LLM.

What Are Tokens?

When you send text to an AI model like Claude, GPT-4, or Gemini, the model does not read words the way you do. Instead, it breaks your text into smaller pieces called tokens. A token might be a whole word, part of a word, a single character, or even a punctuation mark.

Visual Examples

Here is how some common text gets tokenized (using GPT-4's tokenizer as an example):

Notice that spaces are often attached to the beginning of the next token (like " world") rather than being separate tokens. This is a design choice in most modern tokenizers.

Why Tokens Matter

Understanding tokens is not just academic - it has direct practical implications for every AI application you build:

Token-to-Word Ratio

A useful rule of thumb for English text:

This ratio varies significantly by content type:

How Different Languages Tokenize

Tokenizers are typically trained primarily on English text, which means non-English languages often require more tokens to represent the same content:

# Same meaning, different token counts:

English:  "Hello, how are you?"     → ~6 tokens
Spanish:  "Hola, ¿cómo estás?"     → ~8 tokens
Japanese: "こんにちは、お元気ですか？"  → ~12 tokens
Arabic:   "مرحبا، كيف حالك؟"        → ~15 tokens
Chinese:  "你好，你怎么样？"           → ~10 tokens

Tokens in Images and Audio

Modern multimodal models also use tokens for non-text content:

Image Tokens

When you send an image to a vision model, it gets converted into tokens. The number of tokens depends on the image resolution and the model's approach:

• GPT-4o: Images are tiled into 512x512 patches. A typical photo uses 765-1,105 tokens.
• Claude: Images consume tokens based on their pixel count. A 1000x1000 image uses roughly 1,600 tokens.
• Higher resolution = more tokens = higher cost.

Audio Tokens

Audio models (like Whisper or GPT-4o with voice) convert audio into token representations. Speech is typically tokenized at rates of 25-50 tokens per second of audio.