TinkerPreparing the test suite…
Chapter 2 of 7
The test set
Chapter 1’s test set caught a regression nobody tried. Now the team wants more cases in it, and the support lead has a week of the assistant’s chat log to choose from.
This chapter: Which cases belong in a test set, and what does each one need before it can be graded?
You will learn
- Where test cases come from: real questions and known failures
- Why a set needs questions the documents can’t answer
- What a reference answer is, and why every case needs one
Words you’ll meet (4)
- Test case
- One question in the test set, asked on every run, with the answer it should get.
- Reference answer
- The answer a case should get, written by the team. Graders compare the assistant’s answer against it.
- Known failure
- A question the assistant once got wrong. Kept in the set so the same mistake can’t come back unseen.
- Coverage
- Which kinds of case the set includes. A kind with no case is a gap: nothing checks it.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 4
Larkspur’s test set needs more cases. A test case is one question the assistant is asked on every run, with the answer it should give. The support lead exports a week of the assistant’s chat log. Sort it: keep a message as a case, or drop it. Give each kept case its reference answer: the answer a grader will compare the assistant’s against.
“Do you ship orders on Saturdays?”
not sorted“do u ship on weekends”
not sorted“Who pays the shipping when I send something back?”
not sorted“Will my order still ship the same day on Christmas Eve?”
not sorted“Can enterprise customers get a refund?”
not sorted“What is the hotel nightly limit for business travel?”
not sorted“Can I pay in bitcoin?”
not sorted“Will you match a lower price from another supplier?”
not sorted“thanks, that fixed it!”
not sorted