Tinker83 tokens · 37 words · 251 charactersOne paragraph of 47 documents, counted by this site’s tokenizer.
Chapter 3 of 7
Tokens are the meter
Larkspur’s tokenizer learned its pieces from the company’s documents. Now the team wants to know what all those documents cost to send to a model, and whether a quick rule of thumb is good enough to plan with.
This chapter: How many tokens are Larkspur’s documents, and how far off is the rule of thumb?
You will learn
- Why providers bill and size the context window in tokens
- How the “4 characters per token” rule compares with real tokenizers’ counts
- Why the same text costs a different amount on a different tokenizer
Words you’ll meet (3)
- Rule of thumb
- OpenAI's rule of thumb for English: 1 token is about 4 characters, and 100 tokens about 75 words.
- Context window
- The most tokens a model can read in one go: the prompt, the documents and its reply together.
- In-sample
- Counted on the same text the tokenizer was trained on. It looks better than it would on new text.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 4
Larkspur has 47 documents, 47,016 bytes of text. Providers bill and size the context window in tokens, so the team needs a count. OpenAI's rule of thumb for English: 1 token is about 4 characters, and 100 tokens about 75 words. By that rule, 47,016 ÷ 4 is about 11,754 tokens. Two real tokenizers counted the same documents ahead of time. Before you look: