TinkerLoading Larkspur mini
Layer 1 · head 1 · token 1 of 1 (O)Each row of the grid: one token’s attention pairs. Computed live from Larkspur mini.
Chapter 6 of 7
Why long prompts cost more
Larkspur’s finance team sees the bill grow faster than the prompts. Every token in the window looks back at every token before it, and that adds up.
This chapter: Why does the work grow faster than the prompt, and what does the KV cache save?
You will learn
- Why attention pairs grow with the square of the prompt’s length
- What the KV cache keeps so earlier tokens aren’t recomputed
- Why the window has a limit at all
Words you’ll meet (3)
- Attention pair
- One token scoring one earlier token (or itself). Each new token adds one more pair than the token before it, so the pairs grow with the square of the prompt’s length.
- KV cache
- The keys and values of the tokens already read, kept so that each new token only computes its own.
- Context window
- The most tokens a model reads at once. Larkspur mini’s is 32.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 5
In attention, every token scores itself and every token before it. One such score is an attention pair. Open the Attention tab and grow a prompt token by token, to 8 tokens: each new row of the grid is the new token’s pairs.
Loading Larkspur mini…