TinkerBatch cap 4: 192 tokens/s; each token every 18 ms on average; the longest wait for a first token 6.26 s.4 of 4 seats busy · 0 waiting · KV cache 15% full
Chapter 6 of 7
Pace or throughput
Continuous batching helped. Now the team argues about one number: the batch cap, the most requests the server runs at once. Sales wants more customers served; support wants replies that don’t crawl.
This chapter: What does a bigger batch buy, and what does each customer pay for it?
You will learn
- How the batch cap raises throughput and cuts the queue
- Why every request in a bigger batch gets its tokens more slowly
- Why there is no free setting, only a trade-off to choose
Words you’ll meet (2)
- Batch cap
- The most requests the server runs in one pass. Others wait in the queue.
- Queue
- Requests that have arrived but haven’t been let into the batch yet.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 4
Monday’s rush again, with continuous batching and the full KV room. The batch cap is how many requests run in one pass. Each pass reads the weights once, plus each request’s own cache and work, so a fuller pass takes longer. The cap is 4. Before you raise it:
| Batch cap | Throughput | Longest wait for a first token | Time per token, average |
|---|---|---|---|
| 1 | 62 tokens/s | 34.3 s | 15 ms |
| 2 | 113 tokens/s | 15.6 s | 16 ms |
| 4 | 192 tokens/s | 6.26 s | 18 ms |
| 8 | 284 tokens/s | 1.74 s | 22 ms |
| 16 | 321 tokens/s | 187 ms | 26 ms |
Times come from a simulated server: counted work at assumed unit costs, not a measurement of any real server or provider.