TinkerThe toy policy before training: each draft as tall as its odds.
Chapter 3 of 7
Learning from preferences
Your picks from the last chapter are in. Now the model learns from them, without drifting off into answers nobody asked for.
This chapter: How does DPO move a model toward the drafts people prefer, and what holds it back?
About this chapter
You will learn
- How DPO shifts the odds toward preferred drafts
- What the reference policy is
- How the strength β trades your picks against what the model knew
Words you’ll meet
- DPO
- Direct preference optimization: training on preference pairs directly, with no reward model or reinforcement loop.
- Toy policy
- This module’s stand-in for a model: a share of probability for each written draft, not a language model.
- Reference policy
- The model as it was before this training: the starting odds it is pulled back toward.
- Strength
- β: how far the policy may stray from its reference. Low lets your picks dominate; high keeps it close.
Step 1 of 4
trains on preference pairs directly. The here jumps to the result DPO aims for.
Press Train on your picks.
The toy policy is a choice among answers written for this site, not a language model. Its updates run live in your browser.
Direct Preference Optimization (Rafailov and colleagues, 2023) trains on preference pairs with a simple classification loss, with no separate reward model and no reinforcement learning loop. arxiv.org · checked 2026-10-24