TinkerReplace, generation 0: held-out loss 4.04.
Generation 0, replace · held-out loss 4.04 · rare words guessed 27% · all words 44% · repeats 2.6%
Chapter 6 of 7
Keep the real data
Replacing real text with copies drifted. The team ran a second arm that keeps the real documents in every generation.
This chapter: Does keeping the real data in stop the drift?
About this chapter
You will learn
- Replace versus accumulate
- How real data anchors each generation
- Why it slows the loss of rare words but doesn’t stop it
Words you’ll meet
- Replace
- Each generation trains only on the last generation’s text.
- Accumulate
- Each generation trains on the real text plus every generation’s text so far.
- Data mix
- Which texts go into training, and how much of each.
Step 1 of 3
trains each generation only on the last one’s text. keeps the real text and adds each generation’s.
Predict: which stays closer?