TinkerPreparing the test suite…
Chapter 6 of 7
Run it twice
Every chapter so far asked each question once. The team asks a few of them again, and the same question, with nothing changed, comes back passing one time and failing the next.
This chapter: If the same case can pass and fail, what does one run of the test set prove?
You will learn
- Why one try at a case says little about the next
- pass@k and pass^k: “can it?” against “will it, every time?”
- Why a small test set’s score moves in big steps
Words you’ll meet (4)
- Trial
- Model outputs vary between runs, so each attempt at a test case is counted as one trial.
- pass@k
- pass@k is the chance that at least one of k tries is correct.
- pass^k
- pass^k is the chance that all k tries succeed.
- Flaky
- A case that passes on some trials and fails on others, with nothing changed.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 7
Until now each question was asked once. A model can answer the same question differently each time it is asked. Each attempt at a case is called a trial. The team asked 3 cases 5 times each. These recorded answers stand in for asking a live model again and again. Before you look: