What you typed
8 words“Your order of 12 shelves arrives on Friday.”
8 words and 43 characters.
Module 6 · Tokens
What a model reads when you send it text
A model never sees your words. A tokenizer cuts the text into tokens, each one a number, and those numbers are what you pay for and what fill the model’s window. A second kind of number, the embedding, carries meaning, so texts that mean the same sit close together.
Read as
The whole idea in one example
A model reads numbers, not words
What the model reads
21 tokens21 tokens. Tab into the row, then use the arrow keys to read each one.
Each tile is a token: its piece of text over its number.
These are this site’s small tokenizer’s tokens, learned from Larkspur’s own documents. Real tokenizers cut the same sentence differently: GPT-2 into 9 tokens, GPT-4o (o200k_base) into 10 tokens. Their output was recorded ahead of time; this site’s tokenizer runs for real in your browser.
How it works
Three ideas carry the module
1 Tokens, not words
The tokenizer cuts text into numbered pieces
Sennrich and colleagues (2016) brought byte pair encoding to neural translation: rare and unknown words are encoded as sequences of subword units.
Source, checked 21 October 2026
2 Tokens are the meter
Same message in Japanese: 14 tokens on GPT-4o (o200k_base)
Anthropic: usage and billing reflect the model's own tokenizer's counts, so count prompts with the model you plan to use.
Source, checked 21 October 2026
3 Meaning by neighbours
Texts that mean the same sit close together
OpenAI: an embedding is a list of numbers, and the distance between two embeddings measures how related their texts are.
Source, checked 21 October 2026
The story
Larkspur’s assistant starts where the model does
Larkspur Supply Co. is building an AI assistant, and the team starts where the model does: with the tokens it reads and the numbers that carry meaning. You’re on the team that counts what the assistant will read, and what it will cost.
Played RAG? Chapters 5 and 6 read the same map of meaning as RAG’s chapter 5, built from the same embeddings of Larkspur’s documents.
- 47
- Larkspur documents, counted
- 11,754
- tokens by the rule of thumb (bytes ÷ 4)
- 10,242
- tokens on GPT-2
- 9,425
- tokens on GPT-4o (o200k_base)
Your path
The chapters
Chapter 2 · done
How the vocabulary is learned
Where does a tokenizer’s vocabulary come from, and why does it cut words where it does?
Chapter 3 · done
Tokens are the meter
How many tokens are Larkspur’s documents, and how far off is the rule of thumb?
Chapter 4 · done
Every tokenizer splits differently
Why does the same sentence become different tokens on different tokenizers?
Chapter 5 · done
From token to vector
How does text become a list of numbers that carries its meaning?
Chapter 6 · done
Meaning by neighbours
How do you measure how close two meanings are, and find the nearest ones?
Chapter 7 · done
Workbench: fit the prompt
Which parts of Larkspur’s prompt fit in the budget, and which matter most?
Who it’s for
- Product managers
- Engineers
- Analysts
- No code needed
Product managers, engineers and analysts who have used an AI assistant and never looked inside. No background needed; the maths is optional. A good first module.