Larkspur mini
ECE 0.032Rings and mint mostly meet: honest about its own guesses.
Module 30 · Calibration
When can you trust a model’s confidence?
A confidence is a promise: 70% should mean right seven times in ten. Here you drop answers into jars by how sure they claimed to be, measure a real model’s honesty, fix it with one dial, and decide when to hand a customer to a person.
Read as
The whole idea in one example
The same held-out text, before and after a style fine-tune
After a style fine-tune
ECE 0.270Rings above the mint in every jar: overconfident.
Both are Larkspur mini, this site’s tiny model, measured live in your browser on 1,503 places in text it never trained on. Its own numbers, not a large model’s.
How it works
Six ideas carry the module
1 What 70% should mean
Of the answers stated 70%, about seven in ten should be right
2 Honest jars
A tiny model, measured on text it never saw
Kadavath and colleagues at Anthropic (2022) found that larger language models are well calibrated on diverse multiple-choice and true/false questions.
Source, checked 25 October 2026
3 After the polish
A style fine-tune made it overconfident
OpenAI’s GPT-4 report (2023) shows its pre-trained model well calibrated on part of the MMLU test, with an ECE of 0.007, and its post-trained model much less so, at 0.074. OpenAI’s figures, for a far larger model.
Source, checked 25 October 2026
4 One dial
One temperature, fitted on half the text, checked on the other
Guo and colleagues (2017) found that modern neural networks, unlike those of a decade earlier, are poorly calibrated, and that temperature scaling, one number fitted on held-out data, is surprisingly effective at fixing it.
Source, checked 25 October 2026
5 Keeping score
ECE for honesty; the Brier score for honest and useful
Glenn Brier proposed his score in 1950 to check weather forecasts given as probabilities: the average squared gap between each forecast and what happened.
Source, checked 25 October 2026
6 When to hand over
A line below which answers go to a person
Geifman and El-Yaniv (2017) built selective classifiers on trained deep networks: you set the error rate you can accept, and the classifier declines the inputs it must to keep it.
Source, checked 25 October 2026
The assistant’s 60 answers in chapters 1, 6 and 7 are a stand-in written for this site, with hand-set confidences; every other number is measured live.
The story
Larkspur’s support desk
Larkspur Supply Co.’s support team wants to know when to trust its assistant’s confidence, and when to hand a customer to a person. You’re on the team that decides how much to trust each number.
Evals asked whether answers are right; here you ask whether the model knows. The workbench’s best plan is one of 18 that tie.
- 0.032
- Larkspur mini’s ECE on held-out text
- 0.270
- the style fine-tune’s ECE on the same text
- 0.045
- the fine-tune’s ECE on the test half at temperature 1.5
- 23
- hand-overs a day in the workbench’s best plan, of 25 the desk can take
Your path
The chapters
Chapter 1 · done
What 70% should mean
What does a stated confidence promise, and does the assistant keep it?
Chapter 3 · done
After the polish
Does polishing a model’s style change how far its confidence can be trusted?
Chapter 7 · done
Workbench: a dial and a line per kind
What temperature and hand-over line should each kind of question get?
Who it’s for
- Engineers
- Product managers
- Analysts
- No code needed
Product managers, engineers and analysts deciding when a model’s confidence can drive a decision. Evals comes first; no code or maths needed.