Tinker12 test questions, answers included, in the training data.
Chapter 5 of 7
A leaked test
In a third run, 12 of the team’s test questions, answers included, slipped into the training data.
This chapter: Does a model that has seen the test do better at the job?
About this chapter
You will learn
- What a leaked test does to its score
- Why held-out text tells the truth when the test does not
- How to keep a test out of training
Words you’ll meet
- Leak
- Test questions, or their answers, ending up in the training data.
- Contamination
- Another word for a leak: a test is contaminated once the model has trained on it.
Step 1 of 3
A : the test questions sat in the training data. Score the run on them, and on held-out text.
Predict first.