TinkerLarkspur mini’s run: one model size, one data set, one budget.
Chapter 6 of 7
Bigger or longer
The next model gets ten times the compute. The team must choose: a bigger model, or more text to train on.
This chapter: Should extra compute go into a bigger model or into more training data?
About this chapter
You will learn
- What a scaling law is
- How the published advice changed between 2020 and 2022
- Why Larkspur mini’s numbers stay off the published chart
Words you’ll meet
- Compute
- The arithmetic spent on training, roughly 6 × parameters × training tokens operations.
- Scaling law
- A fitted curve that predicts how loss falls as the model, its data or its compute grows.
- Parameters
- Another word for weights. A model’s size is counted in them.
Step 1 of 3
buys or training text. Two papers fit a to many real runs, and disagreed on the split.
Predict first.