Judge the judge

In chapter 3 the model judge passed a made-up policy that a person failed. The team still wants a judge: people can’t read every answer of every run. First they check it.

This chapter: How do you know a model judge grades the way your people would, and what tricks it?

You will learn

How to measure a judge: agreement with people’s labels
How rewriting the rubric changes what the judge catches
Position bias in side-by-side grading, and the fix: judge both orders

Words you’ll meet (4)

Label: A person’s pass or fail on one answer, used to check a grader.
Agreement: How often the judge’s verdict matches people’s labels, out of the answers both graded.
Side-by-side grading: The judge sees two answers to one question and picks the better one. Also called pairwise.
Position bias: Position bias is a model judge's habit of favouring an answer for where it sits, such as first, over what it says.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 7

The model judge is cheap and fast, but is it right? The team labelled every answer the first three versions gave: a person read each one and marked it pass or fail. That mark is a label. Agreement is how often the judge’s verdict matches the label. Before you run the judge over them:

The judge grades every answer the team labelled, with rubric r1: “Is the answer clear, helpful and confident?” How often will it agree with the people?