TinkerStep 3000, the last checkpoint: training loss 2.99.
Chapter 3 of 7
Held out
Training loss fell all run long. But it is measured on the very text the model trained on.
This chapter: Does the model also get better on text it has never seen?
About this chapter
You will learn
- Why you score a model on held-out text
- What the gap between training and held-out loss means
- How memorising shows up as that gap growing
Words you’ll meet
- Held-out text
- Documents kept out of training, used only to score the model.
- Gap
- Held-out loss minus training loss: how much better the model does on text it has seen.
- Memorising
- Learning the training text itself, rather than patterns that carry over to new text.
Step 1 of 4
The team kept some documents back as : never trained on, only used to score the model.
Predict first.