Order for the cache

Each question in a conversation re-sends everything before it: the tools, the instructions, the documents and the history. Most of that is the same as last time.

This chapter: How can Larkspur stop paying full price for the same text on every call?

You will learn

Prompt caching: reusing the start of a recent call’s input
Why one changing line near the top throws the rest away
How to order a turn so the stable parts come first

Words you’ll meet (4)

Prompt caching: The provider keeps the start of a recent input; a call that begins the same way reads it back at a fraction of the price.
Prefix: The start of the input, up to some point. A cache hit needs the prefix to match exactly.
Cache breakpoint: A mark the host puts after a part, saying “cache everything up to here”.
Cache hit: Tokens read back from the cache instead of being sent fresh.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 4

3 questions from Mara Quill in one conversation, a minute apart. Every call re-sends the tool definitions, the instructions, every document and the conversation so far. Prompt caching lets the provider keep the start of a call, its prefix, and read it back on the next call for a fraction of the price, as long as the next call begins exactly the same way. The host marks where a cached prefix may end with cache breakpoints. Send the conversation.