TinkerStatic batching, the busiest pass of Monday’s rush. The last reply is done at 16.5 s.8 of 8 seats busy · 10 waiting · KV cache 29% full
Chapter 3 of 7
Batching
The server can work on several requests in one pass, a batch, because every decode pass reads all the weights anyway. Larkspur’s first server batched the simple way. Monday’s traffic shows the cost.
This chapter: How should a server group requests so none of them waits for the others?
You will learn
- Why batching makes a server serve more for nearly the same cost per pass
- Static batching: a batch starts together and runs until its longest reply ends
- Continuous batching: requests join and leave at every pass
Words you’ll meet (4)
- Batch
- The requests a server runs in the same pass.
- Static batching
- Requests are grouped when the server is free, and the group runs until every reply in it is done.
- Continuous batching
- Also called iteration-level scheduling: after every pass, finished requests leave and waiting ones join.
- Throughput
- Reply tokens the server writes per second, across every request.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 4
Monday’s first rush: 24 requests in five seconds. The server runs up to 8 at once, in a batch: one pass moves every request in it a token on, and reads the weights only once for all of them. It starts with static batching: when it is free it takes up to 8 waiting requests and runs them until the last reply ends. Open the Timeline tab.
| Batching | Last reply done | Longest wait for a first token | Throughput |
|---|---|---|---|
| Static | 16.5 s | 9.78 s | 155 tokens/s |
Times come from a simulated server: counted work at assumed unit costs, not a measurement of any real server or provider.