by Samy (Kondasamy Jayaraman)

Module 38 · Synthetic data

Intermediate · after Fine-tuning vs prompting · by Samy (Kondasamy Jayaraman)

When a model writes its own training data

Larkspur has too few examples of rare requests, so its tiny model writes more. Here you sieve them, test what they teach, and watch copies of copies drift.

Start chapter 1 →See the chapters
Read as

The whole idea in one example

Five generations, two ways

Copies only

gen 5

5.88

Held-out loss, up from 4.04. Rare words guessed: 5%, from 27%.

Real data kept

gen 5

4.33

Held-out loss stays near the start. Rare words guessed: 14%.

Numbers from Larkspur’s toy: Larkspur mini and copies of it, trained offline on Larkspur’s documents. They show the mechanism, not a production model’s figures.

How it works

Four ideas carry the module

  1. 1 Ask the model to write

    A model writes training examples

    Self-Instruct (Wang and colleagues, 2022) had GPT-3 write its own instructions, filtered out invalid or similar ones, and tuned on the rest: a 33% absolute gain on Super-NaturalInstructions.

    Source, checked 25 October 2026

  2. 2 The sieve

    Dedupe and a fact filter drop the weak ones

    phi-1 (Gunasekar and colleagues, 2023), 1.3B parameters, trained on 6B tokens of filtered web text and 1B tokens of GPT-3.5-written textbooks, scored 50.6% on HumanEval.

    Source, checked 25 October 2026

  3. 3 Copies of copies

    Each generation drifts, rare words first

    A large-model finding: Shumailov and colleagues (Nature, 2024) trained models on recursively generated data and found the tails of the original distribution disappear. They call it model collapse.

    Source, checked 25 October 2026

  4. 4 Keep the real data

    Real text in every generation anchors it

    A large-model finding: Gerstgrasser and colleagues (2024) found that replacing real data with each generation’s output leads to collapse, while accumulating it alongside the real data avoids it.

    Source, checked 25 October 2026

The story

Larkspur’s rare requests

Larkspur Supply Co.’s support set has too few examples of rare requests, so the team lets its tiny model write more, and watches copies of copies drift. You’re on the team that trains the assistant.

Played Fine-tuning? It trains on a style set people wrote; here a model writes the set.

42
real documents in Larkspur’s support set
9
rare requests, held out, that every generation is asked
5.88
held-out loss at generation 5 with only copies (4.04 at the start)
4.33
held-out loss at generation 5 with the real documents kept

Your path

The chapters

  1. Chapter 1

    Too few examples

    How often do Larkspur’s documents answer a rare request?

  2. Chapter 2

    Ask the model to write

    What does a model write when you ask it for twenty replies?

  3. Chapter 3

    The sieve

    What does a sieve keep, and what does it throw away?

  4. Chapter 4

    The teacher’s mistakes

    Can a model write good examples of what it never learned?

  5. Chapter 5

    Copies of copies

    What happens when each model trains only on the last one’s writing?

  6. Chapter 6

    Keep the real data

    Does keeping the real data in stop the drift?

  7. Chapter 7

    Workbench: a data recipe

    Which recipe stays closest to real text and still guesses rare words?

Recap: questions from the chapters you’ve finished →

Who it’s for

  • Engineers
  • Product managers
  • Analysts
  • No code needed

Anyone asked to “just generate more training data”, and who wants to know what a model can and can’t teach another. Knowing that a model is trained on examples is enough.