It worked when I tried it

At the end of RAG, Larkspur’s assistant answered staff questions from the company’s own documents. The team keeps tuning it. This week someone changed how big the pieces of each document are, and the answers they tried look better. They want to ship it.

This chapter: How do you know a change made the assistant better, and not just better on the questions you happened to try?

You will learn

Why a change can look better on the questions you try and still break others
What a test set is, and why you run all of it on every change
How to spot a regression by comparing versions question by question, not by their totals

Words you’ll meet (4)

Version: One setup of the assistant: how it cuts the documents, how many pieces it sends the model. Changing a setting makes a new version.
Eval: An eval is a test for an AI system: give it an input, then apply grading logic to its output to measure success.
Test set: A fixed list of questions, each with the answer it should get. Every version runs all of them, so results compare.
Regression: A question that used to pass and now fails, because of a change meant to make things better.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 5

Larkspur’s assistant runs as v1: 32-word pieces, the best 3 sent (RAG’s defaults). The team built v2. Bigger pieces: 48 words; everything else stays the same. A version is one setup of the assistant. Before shipping v2, someone tries a couple of questions on both. Try the two they picked.

Preparing the test suite…