TinkerStatic batching, cap 4, 4 GB of KV room: 4 of 24 requests on target.4 of 4 seats busy · 1 waiting · KV cache 54% full
Chapter 7 of 7
Serve Larkspur’s Monday
Everything is on the bench: the batching mode, the batch cap and how much memory to keep for the KV cache. The support lead has set two targets for every customer.
This chapter: Which serving settings get the most of Monday’s customers their reply on target?
You will learn
- How the settings pull against each other on a real burst of traffic
- How to read a server run request by request
- Why more than one setting can be equally good
Words you’ll meet (1)
- Service target
- A limit the team sets for every request: here, the first token within a second, then each token within a set time.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 2
The support lead’s targets for every customer: the first token within 1.00 s, then each token within 25 ms. The bench starts on Larkspur’s first server: static batching, a cap of 4 and 4 GB of KV room. Change the settings until as many of Monday’s 24 requests as possible are on target.
4 of 24 on target BEST IS 13
| Request | First token | Per token | Result |
|---|---|---|---|
| 1 (420 → 60) | 70 ms | 15 ms | on target |
| 2 (900 → 180) | 1.10 s | 17 ms | late start |
| 3 (260 → 30) | 904 ms | 16 ms | on target |
| 4 (640 → 120) | 664 ms | 16 ms | on target |
| 5 (300 → 220) | 544 ms | 17 ms | on target |
| 6 (1100 → 40) | 4.23 s | 16 ms | late start |
| 7 (380 → 90) | 3.99 s | 16 ms | late start |
| 8 (520 → 150) | 3.87 s | 17 ms | late start |
| 9 (220 → 25) | 3.59 s | 16 ms | late start |
| 10 (760 → 200) | 6.18 s | 17 ms | late start |
| 11 (340 → 70) | 5.98 s | 17 ms | late start |
| 12 (980 → 110) | 5.90 s | 17 ms | late start |
| 13 (280 → 45) | 5.58 s | 17 ms | late start |
| 14 (600 → 240) | 8.94 s | 17 ms | late start |
| 15 (450 → 35) | 8.78 s | 16 ms | late start |
| 16 (820 → 130) | 8.54 s | 16 ms | late start |
| 17 (240 → 80) | 8.38 s | 16 ms | late start |
| 18 (560 → 55) | 12.5 s | 17 ms | late start |
| 19 (1200 → 160) | 12.3 s | 17 ms | late start |
| 20 (320 → 20) | 12.1 s | 17 ms | late start |
| 21 (700 → 190) | 11.9 s | 17 ms | late start |
| 22 (400 → 65) | 15.0 s | 16 ms | late start |
| 23 (880 → 100) | 14.8 s | 16 ms | late start |
| 24 (360 → 140) | 14.5 s | 16 ms | late start |
Times come from a simulated server: counted work at assumed unit costs, not a measurement of any real server or provider.