TinkerLoading Larkspur mini
Chapter 5 of 7
Keeping score
The team wants one number per model to track from release to release, not a row of jars to eyeball.
This chapter: Which single number tells you a confidence can be trusted?
About this chapter
You will learn
- How ECE adds up the gaps between rings and mint
- How the Brier score also rewards being sure and right
- Why a model can be honest and still no help
Words you’ll meet
- ECE
- Expected calibration error: each jar’s gap between stated and actual, weighted by its size. 0 is honest.
- Brier score
- The average squared gap between each confidence and what happened (1 right, 0 wrong). Lower is better.
Step 1 of 3
adds up each jar’s gap between its ring and its mint, weighted by how full it is. 0 is honest.
Predict first.