TinkerLoading Larkspur mini
Layer 1 · head 1 · token 12 of 12 (␣business)The bar: what the sampler draws from at temperature 0, top-p 1. Computed live from Larkspur mini.
Chapter 5 of 7
Choosing the next token
Larkspur mini now gives a probability to every token it knows. Picking the likeliest one every time makes the same reply every time. The team wants to know how a model picks, and what the settings they see in every model’s API change.
This chapter: How do temperature and top-p change which token gets picked?
You will learn
- What sampling is, and why the same prompt can give different replies
- How temperature sharpens or flattens the probabilities
- How top-p keeps only the likeliest tokens before picking
Words you’ll meet (4)
- Sampling
- Picking the next token at random, in proportion to its probability.
- Temperature
- A setting that sharpens the probabilities when low (0 always picks the likeliest) and flattens them when high.
- Top-p
- Nucleus (top-p) sampling, from Holtzman and colleagues, draws from the smallest set of tokens whose probabilities add up to at least p.
- Seed
- The number a random draw starts from. The same seed gives the same draws, so a run can be repeated.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 6
Larkspur mini gives every token a probability. Something still has to pick one. Picking the likeliest every time is called greedy, and it is what a temperature of 0 does. Let it write 8 tokens after “Expedited delivery takes two business”.
Loading Larkspur mini…