Tinker13:00: Larkspur’s shelves after the sale, with chapters 4 and 5’s fixes.9 from shelves · 6 to the depot · 6 trips
Chapter 6 of 7
The same trick everywhere
Larkspur’s shelves keep prices right and the depot calm. The team’s AI assistant runs on a model server. Does it keep copies too?
This chapter: Where else does the keep-a-copy trick show up?
About this chapter
You will learn
- A prompt cache keeps work on a prompt’s opening
- A KV cache keeps a reply’s work for that reply
- The questions every cache answers
Words you’ll meet
- Prompt cache
- A provider’s store of work already done reading a prompt’s opening, reused by matching calls.
- KV cache
- The keys and values a model keeps for tokens it already read, so each new token computes only its own.
Step 1 of 3
Model providers keep copies too. A keeps the work done reading a prompt’s opening.
Predict what breaks a hit.