Tinker9 tokens · 8 words · 43 charactersGPT-2’s tokens, recorded ahead of time.
Chapter 4 of 7
Every tokenizer splits differently
Real tokenizers counted Larkspur’s documents differently from each other. The team now looks at why, sentence by sentence.
This chapter: Why does the same sentence become different tokens on different tokenizers?
You will learn
- How leading spaces and capitals make different tokens
- Why numbers and code cut into odd pieces
- Why other languages often take more tokens for the same meaning
Words you’ll meet (2)
- Leading space
- A space glued to the front of a word’s token: “ shelf” and “shelf” are different tokens.
- Token tax
- The extra tokens one language or kind of text needs, against English, for the same content.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 5
Real tokenizers counted Larkspur’s documents differently in chapter 3. Each one learned its own vocabulary from its own training text, so each cuts the same sentence its own way. The team recorded two real tokenizers, GPT-2 and GPT-4o (o200k_base), on a few fixed sentences. Before you look: