TinkerThe bench before the release gate.
Chapter 6 of 7
Gate the switch
The router saves money at some thresholds and loses answers at others. Before it replaces the all-large assistant, the release gate from the Evals module gets a say.
This chapter: How do you prove a cheaper router is safe to ship?
You will learn
- Why the all-large assistant is the baseline the router must match
- How the gate’s two rules, a pass rate and no regressions, catch a confident wrong answer
- How to find the cheapest setting that still ships
Words you’ll meet (3)
- Baseline
- The version a new one is measured against: here, every question sent to the large tier.
- Regression
- A question the baseline got right that the new version gets wrong.
- Release gate
- The rule that decides, from a test run, whether a candidate ships.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 4
Larkspur ships the all-large assistant today, so it is the baseline. Evals’ release gate has two rules: 90% must pass, and no question all-large gets right may fail. Run the baseline through it first.