TinkerReplace, generation 0: held-out loss 4.04, rare words guessed 27%.
Generation 0, replace · held-out loss 4.04 · rare words guessed 27% · all words 44% · repeats 2.6%
Chapter 5 of 7
Copies of copies
Synthetic text ends up in the next model’s training data, and that model writes more. Larkspur ran it five times over.
This chapter: What happens when each model trains only on the last one’s writing?
About this chapter
You will learn
- What a generation is
- How held-out loss shows drift from real text
- Why rare words go first
Words you’ll meet
- Generation
- One model trained from scratch on the text the one before it wrote.
- Held-out loss
- How surprised a model is by real text it never trained on. Lower is better.
- Drift
- Each generation moving further from the real text.
Step 1 of 4
A is a model trained from scratch on the last one’s writing. Larkspur ran 5, offline.
Predict first.