Release day

The team has a test set, graders it has checked, and a feel for flaky cases. Three new versions wait to replace the one Larkspur ships today. Someone has to decide which, if any, goes out.

This chapter: What rule should decide whether a new version ships, and what can fool it?

You will learn

How a release gate turns a test run into ship or do not ship
Why a pass-rate bar alone lets a regression through
Why the gate is only as good as its grader, and its number of runs

Words you’ll meet (4)

Baseline: The version that ships today. Every candidate is compared against it, question by question.
Candidate: A new version that wants to replace the baseline.
Release gate: The rule that decides, from a test run, whether a candidate ships.
Threshold: The pass rate a candidate must reach to ship, like 75% of the test set.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 3

Release day. v1 is what Larkspur ships today: the baseline, the version every candidate is compared against. A release gate is the rule that decides whether a candidate ships. This one has a threshold, the pass rate a candidate must reach, and a rule that no question v1 passed may fail. It starts at 50% with that rule off, so v2 ships, though chapter 1 showed it breaks a question. Set the gate so it ships v3 and blocks v2.

Candidate

v2: Bigger pieces: 48 words. Baseline v1: 32-word pieces, the best 3 sent (RAG’s defaults).

Grader