Prefill and decode

Larkspur’s assistant is live, and Monday mornings are slow: customers wait before the first word appears, then watch the reply trickle out. The team runs its own model server, so the waiting is theirs to fix. First, what does one request cost?

This chapter: Why does the first token of a reply take a different time from every token after it?

You will learn

Prefill: the model reads the whole prompt in one pass
Decode: the model writes the reply one token per pass
Why time to first token grows with the prompt, while time per token barely does

Words you’ll meet (5)

Forward pass: One run of the model over its input, ending in a score for every possible next token.
Prefill: The first pass of a request: the model reads every prompt token at once and picks the first reply token.
Decode: Every pass after that: the model adds one token to the reply.
Time to first token: How long a customer waits before the reply starts: the queue plus the prefill.
Time per token: How long each later token of the reply takes: one decode pass.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 5

A customer’s request has arrived: “We reply within five business”. Larkspur mini runs in your browser. Press Read the prompt: one forward pass reads all 7 tokens at once and scores the next one. That pass is the prefill.

Loading Larkspur mini