Module 24 · A/B testing

Did the change work, or did you get lucky?

Name: A/B testing
Author: Kondasamy Jayaraman

Split shoppers at random, count who buys, and learn when a gap is real. Then meet the lucky days, early peeks and twenty-colour tests that fake a win, at Larkspur’s shop.

Start chapter 1 →See the chapters

Read as

The whole idea in one example

Two looks at identical checkouts

Peek on day 4

win?

+2.3 pts

Significant, and pure luck.

Read on day 14

no winner

No winner

The planned end keeps no win.

Rates come from simulated shoppers: Larkspur’s own traffic and rates, drawn from fixed seeds, not data from any real shop. The real one-page checkout in chapter 1 ends +2.5 points.

How it works

Four ideas carry the module

1 Split at random
A coin flip decides who sees which checkout
Kohavi et al. (2009): in an A/B test users are randomly exposed to the control or the treatment, and no factor may influence who gets which, so a properly run test’s differences are caused by the change.
Source, checked 24 October 2026
2 Luck makes gaps
Identical checkouts still land apart
Kohavi et al. (2009): an A/A test splits users into two groups but shows both the same experience; at a 95% confidence level it should declare a difference about 5% of the time.
Source, checked 24 October 2026
3 Don’t stop on a good day
An early lead can be luck
Johari et al. (KDD 2017): users of A/B testing software watch results continuously while a test runs, and then the usual p-values and confidence intervals give unreliable inference.
Source, checked 24 October 2026
4 Watch the guardrails
More orders, but more sent back
Kohavi et al. (2009): a good overall criterion is not short-term; with revenue alone you might plaster ads over a page, though many ads hurt the user experience.
Source, checked 24 October 2026

The story

Larkspur tests its checkout

Larkspur Supply Co. tests a new checkout, a bigger button, twenty colours and a pushy banner, and learns which wins are real. You’re the one who reads the results.

Played Evals? It asks the same “is B better?” of an AI assistant.

400: shoppers a day reach each checkout in Larkspur’s simulated shop
14,400: shoppers per checkout to find a one-point lift most of the time
22 of 100: A/A tests call a winner when peeked at every day
37 of 100: runs of twenty identical colours crown one by luck, like Cyan

Your path

The chapters

Chapter 1
Did it work?
How do you tell whether a change worked, and not something else?
Chapter 2
Luck
How big a gap can luck alone make?
Chapter 3
How many shoppers?
How many shoppers does a test need to find a small change?
Chapter 4
Peeking
What does stopping on a good day do to false wins?
Chapter 5
Twenty colours
If you test twenty things, how often does one win by luck?
Chapter 6
The metric moved, the business didn’t
What if the number you tested rises while the business doesn’t?
Chapter 7
Ship, keep testing or stop
Which experiments ship, which keep testing, and which stop?