Module 4 · Evals
Every change to an AI assistant fixes something, and can break something else. Trying a few questions shows the fixes. Running a fixed test set, on every change, shows the breaks too. That is an eval.
The whole idea in one example
The change v2: Bigger pieces: 48 words
“How long does Larkspur keep customer payment card data?”
v2: “Larkspur never stores full card numbers. It keeps only the last four digits and the expiry month, for 90 days after the customer's last order.”
“How long is the warranty on the dock door sensor?”
v2: “The DS-100 dock door sensor is warranted against defects in materials and workmanship for three years from the date of shipment. Batteries are covered for one year.”
Q10, “How much does a Kestrel K2 scanner cost per unit when ordering 50 or more?”, passed on v1 and fails on v2: “The Kestrel K2 handheld scanner is $480 per unit.”
These are the course’s own recorded answers: RAG’s retrieval runs for real on each version, and the answer recorded for what it found is graded against the answer it should get.
How it works
The same questions, every time
An eval is a test for an AI system: give it an input, then apply grading logic to its output to measure success.
Source, checked 20 October 2026
Code, a model judge or a person stamps each answer
Code graders are fast, cheap, objective and repeatable, but brittle: a valid answer that does not match the expected pattern exactly fails.
Source, checked 20 October 2026
Question by question
Model outputs vary between runs, so each attempt at a test case is counted as one trial.
Source, checked 20 October 2026
The story
Back at Larkspur Supply Co., the team keeps changing the assistant, and has to show each change made it better before it ships. You’re on the team that decides whether each change ships.
New to Larkspur’s assistant? The RAG module builds it, and its questions are this module’s test set.
Your path
Chapter 1 · done
How do you know a change made the assistant better, and not just better on the questions you happened to try?
Chapter 2 · done
Which cases belong in a test set, and what does each one need before it can be graded?
Chapter 3 · done
What can grade an answer, and which mistakes does each kind of grader catch or miss?
Chapter 4 · done
How do you know a model judge grades the way your people would, and what tricks it?
Chapter 5 · done
When the assistant takes actions, what should a grader check besides the reply?
Chapter 6 · done
If the same case can pass and fail, what does one run of the test set prove?
Chapter 7 · done
What rule should decide whether a new version ships, and what can fool it?
Product managers, engineers and analysts whose team changes an AI assistant and has to say whether the change made it better. No code needed. If RAG is new to you, play that module first: its questions are the test set here.