TinkerLoading Larkspur mini
Layer 1 · head 1 · token 14 of 28 (␣business)“business” is token 14 and token 28. Numbers from Larkspur mini’s own weights.
Chapter 2 of 7
Tokens become vectors
Larkspur mini picked “days” after “five business”. But a model does arithmetic, and a token is only an id. Before the model can weigh anything, each token has to become numbers.
This chapter: How does a token id become something the model can compute with, and how does it know where the token sits?
You will learn
- What an embedding is: a row of numbers looked up for each token
- Why the same token always starts from the same row
- How adding a position’s row tells the model where each token sits
Words you’ll meet (3)
- Vector
- A list of numbers of fixed length. Larkspur mini’s are 32 numbers long.
- Embedding
- The vector a token is turned into, looked up in a table with one row per token in the vocabulary.
- Position embedding
- A second table with one row per position in the window, added to the token’s row so the model knows where the token sits.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 4
To the model, a token is only an id, and an id is no use for arithmetic. So the first thing Larkspur mini does is swap each id for a vector: a list of 32 numbers. It keeps a table with one row per token in its vocabulary, 768 rows, learned in training. That row is the token’s embedding. Look one up.
Loading Larkspur mini…