TinkerLarkspur mini, a teaching model trained in this course, not any provider’s.
Chapter 1 of 7
Before training
Larkspur’s team has only used models someone else trained. Today they watch one being made: Larkspur mini, trained from nothing on Larkspur’s own documents.
This chapter: What does a language model know before any training at all?
About this chapter
You will learn
- What a model’s weights are before training
- Why an untrained model writes nonsense
- What a checkpoint is
Words you’ll meet
- Pre-training
- Training a model from scratch on a large pile of text, by having it guess each next piece over and over.
- Weights
- The numbers inside the model that decide what it writes. Before training they are random.
- Checkpoint
- A saved copy of the weights at one step of training, like a photo of the model at that moment.
- Next piece
- What a language model guesses, one at a time: the next token of the text (see the Tokens module).
Step 1 of 5
starts here. Larkspur mini’s are random numbers, saved at step 0 as the first .
Open the step-0 checkpoint.