TinkerThe same checkout on both sides.
Chapter 2 of 7
Luck
The new checkout won by about two points. Then someone asks: would two identical checkouts differ too? The team pits today’s checkout against itself.
This chapter: How big a gap can luck alone make?
About this chapter
You will learn
- An A/A test shows the noise between identical groups
- What significant means
- Why a few A/A tests still call a winner
Words you’ll meet
- A/A test
- A test of a version against an identical copy of itself: any gap is luck.
- Noise
- Gaps that come from which shoppers happened to buy, not from any change.
- Significant
- A gap bigger than luck alone makes 19 times in 20 (a 5% level).
- False win
- A significant result when nothing truly differs.
Step 1 of 5
An splits shoppers between two identical checkouts. Any gap is : luck in who happened to buy.
Run the A/A test.