Reading the prompt
1 pass“We reply within five business”
7 tokens, read together: the first token after 16 ms.
Module 13 · Inference
What happens between the prompt and the reply
A model server reads each prompt in one pass, then writes the reply one token per pass, for many customers at once. Here you run one, from a single request to Monday morning’s rush: batching, the KV cache, smaller numbers and a small model that guesses ahead.
Read as
The whole idea in one example
One request, two speeds
Writing the reply
6 more passes6 more tokens, one pass each
About 15 ms a token: 91 ms in all.
Times are from a simulated server running OPT-13B’s published shape: counted work at assumed unit costs, not a measurement of any real server. The tokens are Larkspur mini’s, run for real in your browser in chapter 1.
How it works
Four ideas carry the module
1 The KV cache
Every request in flight holds its own cache
For OPT-13B, the KV cache of a single token takes 800 KB: 2 (key and value) × 5,120 × 40 layers × 2 bytes. A request of 2,048 tokens can need 1.6 GB (Kwon et al., 2023).
Source, checked 21 October 2026
2 Batching
One pass moves every request in the batch a token on
Orca introduced iteration-level scheduling: the server schedules one model iteration at a time, so finished requests leave the batch and new ones join between iterations. On GPT-3 175B it reported 36.9× the throughput of NVIDIA FasterTransformer at the same latency (Yu et al., OSDI 2022).
Source, checked 21 October 2026
3 Smaller numbers
Rounded weights leave room for more requests
GPTQ quantizes the weights of a 175-billion-parameter GPT model to 3 or 4 bits in about four GPU hours, with negligible loss of accuracy (Frantar et al., ICLR 2023).
Source, checked 21 October 2026
4 Guess ahead
A small model drafts, the big one checks
Speculative decoding computes several tokens in parallel by checking a smaller model’s guesses with the large one, without any change to the outputs; on T5-XXL it ran 2–3× faster with identical outputs (Leviathan et al., 2023).
Source, checked 21 October 2026
The story
Larkspur serves its own model
Back at Larkspur Supply Co., the team now serves its own model, and has to keep Monday morning’s customers from waiting. You’re on the team that runs the server.
Played Transformers? Chapter 2 picks up its KV cache from chapter 6 and multiplies it over a real server. OPT-13B’s shape: 40 layers, d_model 5,120.
- 800 KB
- of KV cache per token on OPT-13B, counted from its shape
- 24
- requests in Monday’s rush, the trace every server chapter runs
- 51,072
- numbers in Larkspur mini, the model that runs in your browser
- 16,112
- in its student, the draft model of chapter 5
Your path
The chapters
Chapter 1 · done
Prefill and decode
Why does the first token of a reply take a different time from every token after it?
Chapter 2 · done
The KV cache at scale
How much memory does the KV cache take on a real server, and what does that limit?
Chapter 4 · done
Smaller numbers
What does storing the weights in fewer bits save, and what does it cost?
Chapter 5 · done
Guess ahead
How can a small model make a big one faster without changing a word of its reply?
Chapter 6 · done
Pace or throughput
What does a bigger batch buy, and what does each customer pay for it?
Chapter 7 · done
Serve Larkspur’s Monday
Which serving settings get the most of Monday’s customers their reply on target?
Who it’s for
- Engineers
- Product managers
- Analysts
- No code needed
Engineers, product managers and analysts who run or pay for a model in production and want to know where the waiting goes. It builds on Transformer architecture; any maths is optional.