TinkerHalf the documents, stopped at step 3000: held-out loss 4.58.
Chapter 7 of 7
Larkspur’s training run
The team gets one more training run before launch. They choose the data, and when to stop.
This chapter: Which data and which stopping step give the best score on text the model never saw?
About this chapter
You will learn
- How the data mix and the stopping step trade off
- Why a small data set peaks early
- Why more than one plan can tie for best
Words you’ll meet
- Stopping step
- The training step whose checkpoint you keep and ship.
- Early stopping
- Keeping an earlier checkpoint because held-out loss was lowest there, before memorising took over.
Step 1 of 2
Each mix is a real recorded run. Its score is held-out loss at your : lower wins.
Find the best plan.
4.58 held-out loss BEST IS 4.04
| Data mix | Held-out loss |
|---|---|
| All documentsEvery document once, stopped where held-out loss had stopped falling. | 4.04 |
| One document ×10One document was copied 10 times, so the model spent its steps memorising it. | 4.19 |
| Test questions inThe test questions it trained on do nothing for text it has never seen. | 4.08 |
| Half the documents (yours)Half the documents: held-out loss was lowest at step 600; after it, the model memorised. | 4.58 |