What training costs you

Training taught Larkspur mini a sign-off. The team wonders what else it changed, on everything that was not in the style set.

This chapter: What does a model lose when it is fine-tuned, and does a small adapter lose less?

You will learn

How to measure forgetting: score text the model never trained on, before and after
Why a full fine-tune can do worse on the text it used to know
How LoRA trades learning less for forgetting less

Words you’ll meet (2)

Held-out text: Text kept out of training, used only to score the model.
Forgetting: Catastrophic forgetting is a neural network losing what it learned before when it is trained on something new; Kirkpatrick and colleagues note it was long thought to be unavoidable.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 4

Training taught Larkspur mini a sign-off. What did it cost? To find out, score text the model never trained on: held-out text. Larkspur kept 5 of its documents out of every training run. The score is the loss from chapter 2, averaged over all 1508 pieces: lower means the model predicts the text better. Score the base model first.

Held-out documents

data-retention-faq-2025.md, heron-shelving-install-guide-2021.md, portal-release-notes-2021.md, remote-work-policy-2021.md, warehouse-safety-handbook-2023.md

Loading Larkspur mini