How close is the student?

The student trained on the teacher’s odds. Before Larkspur trusts it with anything, the team wants to know how often it still picks what the teacher would.

This chapter: How do you measure how close a student is to its teacher, on text neither has seen?

You will learn

Agreement: how often the two models’ top picks match
KL divergence: how differently they spread their bets, even when the top pick matches
Why closeness to the teacher is not the same as being right

Words you’ll meet (3)

Held-out text: Text kept out of training, used only to score the models.
Agreement: The share of positions where the student’s most likely next piece is the teacher’s.
KL divergence: A number for how far one set of odds is from another: 0 when they are the same, larger the more they differ.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 3

The student copied the teacher’s odds on the training text. The fair test is text neither model trained on: held-out text. At every place in it, both models pick their most likely next piece. Before you run them:

Held-out documents

data-retention-faq-2025.md, heron-shelving-install-guide-2021.md, portal-release-notes-2021.md, remote-work-policy-2021.md, warehouse-safety-handbook-2023.md

Run both models over 5 of Larkspur’s documents that neither trained on: 1,503 places to guess the next piece. How often will the student’s top pick be the teacher’s?