TinkerRewards under the weak grader: A 0, B 1, C 1, D 0, E 2.
Chapter 6 of 7
You get what you check
Checking key facts means writing references. A teammate proposes a cheaper grader: reward any reply with a number and Larkspur’s sign-off.
This chapter: What does a model learn from a grader that checks the wrong thing?
About this chapter
You will learn
- What reward hacking looks like
- Why a weak grader’s favourite can be wrong
- How a better grader fixes it
Words you’ll meet
- Weak grader
- A grader that checks something easy, like the presence of a number, instead of whether the answer is right.
- Reward hacking
- Scoring high on the grader without doing the job the grader stands for.
Step 1 of 4
A : a point for a number, a point for the sign-off. Answer E has both, and the wrong refund window.
Predict first.
Answer B written for this site
“Standard customers can request a refund within 30 days of delivery.”
PASS has the key factPASS has a numberFAIL signs off
Right, and word for word the reference.
Answer E written for this site
“Refunds take 7, 14 or 60 days, it depends. Cheers, the Larkspur team”
FAIL has the key factPASS has a numberPASS signs off
Games a weak grader: numbers and a sign-off, but not the 30 days.
The toy policy is a choice among answers written for this site, not a language model. Its updates run live in your browser.