TinkerSaved addresses: 14 days, 5,600 shoppers per checkout.day 14 of 14 · A 10.6% · B 12.0% · A returns 5.1% · B returns 3.9%
Chapter 7 of 7
Ship, keep testing or stop
Five experiments finished this month, each run to its planned end with no peeking. The owner wants a decision on each.
This chapter: Which experiments ship, which keep testing, and which stop?
About this chapter
You will learn
- Apply one rule to every experiment
- Why underpowered is not the same as no effect
- Why a rule can still ship a lucky win
Words you’ll meet
- Decision rule
- A rule written before the results come in, saying what each kind of result leads to.
Step 1 of 2
The : ship if significant and returns hold; keep testing if underpowered; otherwise stop.
Decide all 5.
1 of 5 follow the rule NOT YET
A test needs 14,400 shoppers per checkout to find a 1-point change most of the time.
Buying +1.4 points (+0.3 to +2.6): significant · returns hold · 5,600 shoppers each (underpowered)
Buying +0.4 points (−0.7 to +1.6): not significant · returns hold · 5,600 shoppers each (underpowered)
Buying +1.5 points (+0.4 to +2.7): significant · returns up · 5,600 shoppers each (underpowered)
Buying +0.1 points (−0.6 to +0.7): not significant · returns hold · 16,800 shoppers each
Buying +1.0 points (+0.4 to +1.7): significant · returns hold · 16,800 shoppers each
Simulated shoppers: Larkspur’s own traffic and rates, drawn from fixed seeds, not data from any real shop.