TinkerThe gate, ready: three kinds of question, four efforts.
Thinking recorded
Run a question to see what the model wrote before it answered.
Chapter 6 of 7
Measure, then pick
Every kind of question Larkspur gets has its own best effort, and nobody wants to guess. The release gate from the Evals module gets a say.
This chapter: How do you choose an effort for each kind of question, with numbers?
You will learn
- How to run a sweep through a release gate, one kind of question at a time
- Why the right setting is the lowest effort that passes, not the highest
- How the no-regressions rule catches overthinking
Words you’ll meet (3)
- Release gate
- The rule that decides, from a test run, whether a change ships. Here: a pass-rate bar, and no question that passed without thinking may fail.
- Baseline
- What a change is measured against: here, answering with no thinking.
- Regression
- A question that used to pass and now fails.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 3
A release gate is the rule that decides, from a test run, whether a change ships; this is the one from the Evals module, unchanged. Here a kind of question ships at an effort if 80% or more of its answers pass and none that passed with no thinking (the baseline) now fails: a question that used to pass and now fails is a regression. Before you run it:
- Policy lookups (4) One fact, read straight off a document. Questions from the Evals module’s test set.
- Multi-step questions (6) Quotes with volume prices, stock across warehouses, delivery dates: several steps, each easy to slip on.
- Simple questions with distractions (3) An easy count, wrapped in details that have nothing to do with it.