Guess ahead

Each decode pass writes one token and reads every weight to do it. Larkspur has a much smaller model, a student trained to copy Larkspur mini. What if the student guessed the next few tokens, and the big model only checked them?

This chapter: How can a small model make a big one faster without changing a word of its reply?

You will learn

Speculative decoding: a draft model guesses, the target model checks every guess in one pass
Why the reply is exactly what the target would have written alone
Acceptance rate and tokens per check, and why longer guesses stop paying

Words you’ll meet (4)

Draft model: A small, fast model that guesses the next few tokens.
Target model: The model whose reply you want; it checks the draft.
Speculative decoding: Drafting k tokens with a small model and checking them with the target in one pass, keeping the ones it agrees with.
Acceptance rate: The share of drafted tokens the target agrees with.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 5

Each decode pass of Larkspur mini writes one token and reads every weight to do it. Write 12 tokens after “Questions go to the billing team through” the plain way, one pass each.

Loading Larkspur mini