Step 0 run at build time
“Returns are accepted within…”
dock dock dock dock dockiiiiiiiiiiiiiiiiiii
Random weights: noise.
Module 18 · Pre-training
Where does a model come from?
Every model starts as random numbers and learns by guessing the next piece of text, over and over. Here you open a tiny model’s real training run, checkpoint by checkpoint, and see what its data teaches it.
Read as
The whole idea in one example
One prompt, before and after training
Step 3,000 run at build time
“Returns are accepted within…”
five business days.
After the whole run: Larkspur’s kind of sentence.
Both replies come from Larkspur mini’s real checkpoints, greedy, the same way the chapters run them in your browser. Nothing changed between them but the weights.
How it works
Six ideas carry the module
1 From nothing
Random weights, then thousands of small nudges
Kaplan and colleagues (2020) found that a language model’s loss falls as a power law of its size, its training data and the compute spent on it.
Source, checked 24 October 2026
2 Checkpoints
Saved copies let you look back at any moment of a run
3 Held out
Score the model on text it never trained on
4 Repeats get memorised
Copied text comes back word for word
Lee and colleagues (2021) found that over 1% of the unprompted output of models trained on common datasets was copied word for word from the training data.
Source, checked 24 October 2026
5 A leaked test flatters
A test the model trained on stops measuring
Magar and Schwartz (2022) trained models with test data mixed into pre-training and found they sometimes exploit it, scoring higher on the examples they saw than on ones they didn’t.
Source, checked 24 October 2026
6 Size and data together
More compute means a bigger model and more text
Hoffmann and colleagues (2022) found that model size and training tokens should be scaled equally: for every doubling of model size, double the training tokens too.
Source, checked 24 October 2026
The chapters run Larkspur mini’s saved checkpoints in your browser; the other runs were trained ahead of time and recorded. Its numbers teach the mechanism; they are not any provider’s.
The story
Larkspur trains its own
Back at Larkspur Supply Co., the team trains its own tiny model from scratch, and learns what the data, the test set and the stopping point do to it. You’re on the team that picks the data and decides when to stop.
Tokens built the pieces a model reads. This module trains a model on them.
- 51,072
- weights in Larkspur mini
- 42
- Larkspur documents it trains on
- 3,000
- training steps in its run
- 8
- checkpoints you can open
Your path
The chapters
Chapter 6 · done
Bigger or longer
Should extra compute go into a bigger model or into more training data?
Chapter 7 · done
Larkspur’s training run
Which data and which stopping step give the best score on text the model never saw?
Who it’s for
- Engineers
- Product managers
- Analysts
- No code needed
Product managers, engineers and analysts who want to know what “trained on” means. Tokens comes first; no code or maths needed.