Tinker Small models: distillation & routing

by Kondasamy Jayaraman · drawn with Colorfigures by April Zhu

OK

Policy score 13.6: lookups → router; policy questions → router; vague or multi-step → router, router up below 0.9.

Chapter 7 of 7

Larkspur’s routing policy

One threshold for every question works, but the questions are not all alike. The support lead sorts them into kinds and asks for a policy per kind: small, router or large.

This chapter: Which tier should answer each kind of question?

You will learn

  • How to match a tier to what a kind of question needs
  • How right answers and cost pull against each other in one score
  • Why the router suits some kinds and not others
Words you’ll meet (2)
Routing policy
A rule for each kind of question: always small, always large, or the router.
Policy score
Points for every right answer, minus what the answers cost, added up over every kind.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 2

Chapter 6’s router ships with one threshold for every question. The support lead sorts the questions into kinds and wants a policy per kind. The bench scores a policy: 2 points per right answer, minus the cost in large answers. The team values a right answer at two large-model answers: a wrong one sends the customer to a person, which costs more than the call. The answers are recorded; the grading runs live. Find a policy with the best score.

One fact, stated in one line of one document.

Rules with exceptions, or old and new versions that disagree.

Questions that need a guess at what was meant, or two hops.

13.6 points BEST IS 15.2

11 of 12 right · cost 8.4

Each question under your policy
QuestionAnswered byKey fact
How long do standard customers have to ask for a refund?Short and right: one line in one document.small answeredconfidence 0.95PASS
Can I open a dock door when its sensor shows a red light?Short and right: the handbook says it in one line.small answeredconfidence 0.93PASS
What is the cutoff time for same-day shipping?Short and right: one line in the shipping page.small answeredconfidence 0.94PASS
How much does a Kestrel K2 scanner cost per unit when ordering 50 or more?Right: the same price the small tier read.sent up, large answeredconfidence 0.74PASS
How long is the warranty on the dock door sensor?Short and right: one line in the warranty sheet.small answeredconfidence 0.91PASS
Who leads the Customer Success team?Short and right: a name the team page states.small answeredconfidence 0.97PASS
Can enterprise customers get a refund?Short and right: the exclusion is one plain sentence in the refund policy.small answeredconfidence 0.92PASS
What is the hotel nightly limit for business travel?Right: it noticed two policies and picked the newer one.sent up, large answeredconfidence 0.84PASS
How long does Larkspur keep customer payment card data?Right: it told the demo cards apart from customers' cards.sent up, large answeredconfidence 0.55PASS
it's broken again, who do I even call?Right: it read the vague question as an outage and found the out-of-hours number.sent up, large answeredconfidence 0.42PASS
What payment methods does the Portal accept?Wrong for both tiers: the digital-wallet line sits in a passage retrieval never sends, so no model size helps.sent up, large answeredconfidence 0.62FAIL
Who approves purchases for the team that owns the Kestrel K2 scanner integration?Right: it made both hops, team then approver.sent up, large answeredconfidence 0.66PASS