What the model reads

You have joined the team building an AI assistant for Larkspur Supply Co., a hardware company. Before the assistant reads a single document, the team wants to know what a model actually reads when you send it text.

This chapter: When you type a sentence, what does the model receive?

You will learn

Text reaches the model as a list of tokens, each one a number
Why tokens are not words and not letters
How to count a sentence in tokens, words and characters

Words you’ll meet (3)

Token: A piece of text the model reads as one unit: a word, part of a word, a space with a word, or a single byte. Each token has a number, its id.
Tokenizer: The program that cuts text into tokens and turns them into ids, before the model sees anything.
Vocabulary: Every token a tokenizer knows, each with its id. Text can only be cut into tokens from this list.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 4

Larkspur’s assistant will read messages like this one: “Your order of 12 shelves arrives on Friday.” A model can’t read letters. Before it sees anything, a tokenizer cuts the text into tokens and turns each into a number. Here the tokenizer is this site’s own, trained on Larkspur’s documents. Before you feed the sentence in:

How many tokens will the tokenizer cut “Your order of 12 shelves arrives on Friday.” into? It has 8 words.