Learning from a teacher

The small model in chapter 1 was not trained from scratch. It learnt from Larkspur mini itself, by copying how the bigger model spreads its bets.

This chapter: How does a small model learn from a big one, and what does it learn that the text alone doesn’t teach?

You will learn

The difference between the right answer and the teacher’s odds
What a training run of distillation looks like, step by step
Why the student ends a little behind the teacher, not far behind

Words you’ll meet (4)

Distillation: Hinton, Vinyals and Dean train a small model on the class probabilities a large model produces, used as “soft targets”, raising the temperature of the large model’s final softmax until those targets are suitably soft.
Teacher and student: The big model whose odds are copied, and the small model that learns to copy them.
Soft targets: The teacher’s whole set of odds over the next piece, used as the lesson instead of the one right piece.
Loss: How surprised a model is by the next piece of text, on average. Training lowers it.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 4

Training a model from text alone teaches one right next piece at a time. Distillation teaches the teacher’s whole set of odds instead, softened at temperature 2 so the near-misses count, with a small weight (0.1) left on the right piece. Open one lesson.

Loading Larkspur mini