Module 19 · Post-training
After pre-training comes post-training. Here you pick the better of two drafts, train a toy policy on your picks, let a grader mark its drafts, and write a grader it can’t game.
The whole idea in one example
“How long do standard customers have to ask for a refund?”
Q: How long is the warranty on a dock sensor? Q: Can I return a ladder? Q: Where is my order?
A base model’s continuation: it carries on the FAQ page instead of answering.
“How long do standard customers have to ask for a refund?”
Standard customers can request a refund within 30 days of delivery.
Right, and word for word the reference.
Both answers were written for this site; the marks come from the Evals module’s key-fact check, run at build time. The module’s toy policy is a choice among written answers, not a language model: its updates run live and teach the mechanism, not any provider’s numbers.
How it works
Examples teach a base model to answer
InstructGPT (Ouyang and colleagues, 2022) was made in three stages: fine-tuning GPT-3 on answers people wrote, collecting people’s rankings of the model’s outputs, then reinforcement learning from that human feedback.
Source, checked 24 October 2026
People pick the better of two drafts
In the InstructGPT paper, people preferred the answers of the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3, a model about 100 times bigger.
Source, checked 24 October 2026
DPO leans toward them, held near the reference
The DPO paper shows the policy it aims for: the reference model’s odds, scaled up by the reward, with a parameter β controlling how far it may stray from the reference.
Source, checked 24 October 2026
A program checks each draft against its group
Group Relative Policy Optimization (DeepSeekMath, 2024) drops PPO’s critic model: it samples a group of answers per question and scores each against the group’s average reward, divided by the group’s spread.
Source, checked 24 October 2026
Equal rewards move nothing
Tülu 3 (2024) named Reinforcement Learning with Verifiable Rewards: the model gets a reward only when its answer is verified correct, on tasks such as maths and checkable instructions.
Source, checked 24 October 2026
A weak grader trains the gaming answer
DeepMind (2020) calls it specification gaming when an agent meets the literal objective without the intended outcome, like a boat that circles hitting reward targets instead of finishing its race.
Source, checked 24 October 2026
Each stack of cards is one written answer, as tall as the toy policy’s odds of giving it. The training runs in your browser with fixed seeds, so every visit shows the same rounds.
The story
Back at Larkspur Supply Co., the team is turning a base model into its assistant: with people’s picks, with graders, and by finding where a grader gets gamed. You’re on the team that chooses what the model learns from, and checks that it learns the right thing.
Fine-tuning trained Larkspur mini on examples; Evals built the key-fact check every grader here uses.
Your path
Chapter 2 · done
How do people’s preferences become something a model can learn from?
Chapter 3 · done
How does DPO move a model toward the drafts people prefer, and what holds it back?
Chapter 5 · done
What happens when a grader can’t tell the good drafts from the bad ones?
Chapter 6 · done
What does a model learn from a grader that checks the wrong thing?
Chapter 7 · done
Which rules make a grader that rewards right answers and nothing that games it?
Product managers, engineers and analysts who read “trained with RL” in a launch post and want to know what it means. Fine-tuning and Evals come first; no code or maths needed.