From trace to test

The team rebuilt the policy index so it holds only this year’s passages. Did that fix the old-passage failures, and will it stay fixed?

This chapter: How do failed traces become tests that guard the fix?

You will learn

Turning a failed trace into an eval case with a reference answer
Running the cases on the current and the fixed version, with Evals’ graders
Why the cases join the suite for good

Words you’ll meet (2)

Eval case: An input with a reference answer and a grader, as in the Evals module.
Regression: Something that worked, or was fixed, breaking again later. A test from a real failure catches it.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 5

The team shipped a fix for “Answered from an old passage”. To check it, and to keep it fixed, each failed trace can become an eval case: its question, a reference answer, and Evals’ graders. Add every “answered from an old passage” trace as a case; add others if you like.

In Evals you graded a suite written before launch. Here the cases come from what customers really asked.

Quarry Lane Builders: “What time do I have to order by for same-day shipping?”
Answered from an old passage · Quoted the 2024 cutoff, not this year’s
Oak & Iron: “How long will you keep my card on file?”
Answered from an old passage · Gave last year’s card rule
Birch & Co: “What’s the warranty on the DS-100 dock sensor?”
Answered from an old passage · Old warranty sheet: one year instead of three
Marlow Builders: “Price for 60 Kestrel K2 scanners?”
Answered from an old passage · Quoted an old price for the K2
Hollis Hardware: “How long do I have to send something back?”
Answered from an old passage · Said 14 days; the policy says 30
Fenwick Tools: “The Portal is down and it’s 10 p.m. Who do I call?”
Search missed the passage · Search missed the on-call guide
Marlow Builders: “Can I pay an invoice by bank transfer?”
Search missed the passage · Couldn’t find the payments policy
Oak & Iron: “A dock door light is red. Can I open it?”
Search missed the passage · No safety card retrieved; gave up
Kestrel Logistics: “Who runs Customer Success now?”
Search missed the passage · Directory not found, said it didn’t know
Quarry Lane Builders: “Who signs off K2 integration purchases?”
Search missed the passage · Wrong passages for an approver question
Hollis Hardware: “Do you take Apple Pay or other wallets?”
Search missed the passage · Missed the wallets line, answered nothing
Hollis Hardware: “Has order 1042 arrived?”
Looked up the wrong order · Looked up 1024 when they said 1042
Hollis Hardware: “Where is order 1043?”
Looked up the wrong order · Swapped the digits of the order number
Birch & Co: “Did order 1024 arrive?”
Looked up the wrong order · Answered about another customer’s order
Hollis Hardware: “Is order 1043 on its way?”
Ignored what the tool returned · Said shipped; the order says processing
Birch & Co: “When did order 1024 get here?”
Ignored what the tool returned · Called it in transit; it was delivered
Kestrel Logistics: “Can we get a refund on last month’s scanners?”
Promised a refund it can’t give · Promised an enterprise refund it can’t give
Marlow Builders: “Our shelving order was late. Refund it, please.”
Promised a refund it can’t give · Said it refunded; it can’t, and didn’t