Tinker1 request of 2,048 tokens: 1.7 GB of cache in a 14 GB room.1 of 8 seats busy · 0 waiting · KV cache 12% full
Chapter 2 of 7
The KV cache at scale
Decoding is fast because the model keeps every token’s keys and values instead of recomputing them. Larkspur mini’s cache is tiny. The team’s server runs a real open model, and serves many customers at once.
This chapter: How much memory does the KV cache take on a real server, and what does that limit?
You will learn
- The KV cache, from the Transformers module, in one step
- Its size for a real open model, from its published shape
- Why memory, not compute, caps how many requests a server can hold at once
Words you’ll meet (3)
- KV cache
- The keys and values the model keeps for every token it has read, so each new token computes only its own.
- KV room
- The memory left for the KV cache once the model’s weights are loaded.
- In flight
- Requests the server is working on right now, each holding its own cache.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 5
In the Transformers module you grew a prompt token by token and watched the KV cache grow: every layer keeps a key and a value for every token, so each new token computes only its own. Count Larkspur mini’s cache for a full window.