Graders

Larkspur’s test set is built, and every case has a reference answer. Now something has to decide, for each answer, pass or fail.

This chapter: What can grade an answer, and which mistakes does each kind of grader catch or miss?

You will learn

Three code checks: exact match, contains the key fact, has a citation
What a model judge and its rubric are, and how a rubric can be fooled
Why people are the best grader and the slowest

Words you’ll meet (6)

Grader: Whatever decides whether one answer passes: a code check, a model, or a person.
Exact match: A code check: the answer’s words equal the reference answer’s.
Key fact: The short phrase an answer must contain to be right, like “30 days”.
Citation: The passage from the documents an answer says it rests on.
Model judge: Another model that reads an answer and grades it, pass or fail.
Rubric: The written test a model judge applies: the question it answers about each answer.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 7

A grader decides whether one answer passes. Larkspur has three kinds. Code checks run on the words: exact match (are they the reference answer’s words?), contains the key fact (is the short phrase that matters there, like “30 days”?), and has a citation (does it name the passage it rests on?). A model judge is another model that reads the answer against a rubric, a written test it answers pass or fail. Its rubric here, r1: “Is the answer clear, helpful and confident?” And a person from the team reads it. Here is answer 1. Before you grade it:

Preparing the test suite…

Answer 1 is right, in its own words. Which grader fails it anyway?