Tinker A/B testing

by Kondasamy Jayaraman · drawn with Colorfigures by April Zhu

Checkout B shows an “Only a few left!” banner.

Chapter 6 of 7

The metric moved, the business didn’t

A banner says “Only a few left!” and orders jump. The team is ready to ship, until the returns desk calls.

This chapter: What if the number you tested rises while the business doesn’t?

About this chapter

You will learn

  • The primary metric is what the test is for
  • Guardrail metrics watch what must not get worse
  • Ship only when the guardrails hold too

Words you’ll meet

Primary metric
The one number a test is run to move: here, the share who buy.
Guardrail metric
A number that must not get worse, like returns, checked before anything ships.

Step 1 of 5

The is the number a test is run to move: here, the share who buy.

Run the banner test.