Tinker A/B testing

by Samy (Kondasamy Jayaraman) · drawn with Colorfigures by April Zhu

20 colours, one turnstile.

Chapter 5 of 7

Twenty colours

A designer proposes twenty colours for the Buy button and tests every one against today’s blue at once.

This chapter: If you test twenty things, how often does one win by luck?

About this chapter

You will learn

  • Every extra comparison is another chance of a false win
  • Why a lone winner among many needs doubt
  • A correction raises the bar for each comparison

Words you’ll meet

Multiple comparisons
Running many tests at once, each one another chance for luck to win.
Correction
A stricter bar for each test, so the whole family keeps its false wins near the level.

Step 1 of 5

None of the 20 colours truly changes buying; the simulation knows. Each is compared with blue.

Run the colour test.