TinkerPlan score 2: replies with no facts in them → Fine-tuned; questions about policies and products → Fine-tuned; prices that just changed → Fine-tuned.
Chapter 7 of 7
Larkspur’s decision
Everything is on the bench: the prompted, RAG and fine-tuned versions, with their scores from the release gate. The support lead needs a plan for each kind of request.
This chapter: Which lever should Larkspur pull for each kind of request?
You will learn
- How to match a lever to what the request needs: a style, a fact, or a skill
- How quality, cost and upkeep pull against each other
- Why more than one plan can be equally good
Words you’ll meet (2)
- Upkeep
- The work a choice needs after launch: retraining, refreshing documents, re-running the evals.
- Plan score
- Quality from the graded answers, minus cost and upkeep, added up over every kind of request.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 2
The support lead’s brief: “Just fine-tune it” sends every kind of request to the fine-tuned version. The bench scores a plan. Quality is up to 6 points per kind: the share of its recorded answers that pass, graded live (the key fact, or the sign-off for replies with no facts). Then subtract cost and upkeep, the work a choice needs after launch: those come from the team’s own notes, not measured facts. Send each kind to one version and find a plan with the best score.
2 points BEST IS 10
| Kind, sent to | Quality | Cost | Upkeep | Score |
|---|---|---|---|---|
| Replies with no facts in them: Fine-tuned13 of 13 pass. Cost: One training run and a round of evals; nothing extra per request. Upkeep: The voice rarely changes, so it is rarely retrained. | 6 | −1 | 0 | 5 |
| Questions about policies and products: Fine-tuned6 of 12 pass. Cost: One training run and a round of evals; nothing extra per request. Upkeep: Retrain and re-run the evals whenever a fact changes. | 3 | −1 | −2 | 0 |
| Prices that just changed: Fine-tuned0 of 1 pass. Cost: One training run and a round of evals; nothing extra per request. Upkeep: Retrain and re-run the evals whenever a price changes. | 0 | −1 | −2 | −3 |
| Plan | 9 | −3 | −4 | 2 |