TinkerRouter, up below 0: 7 of 12 right, cost 2.4, 0 sent up
- router, one point per threshold
- all large
Up below 0: 7 of 12 right, cost 2.4, 0 sent up. “What is the hotel nightly limit for business travel?” stays small.
Chapter 5 of 7
Fall back when unsure
The small tier gets the easy questions right and stumbles on the tricky ones. The team wires a router: the small tier answers first, and when it is unsure, the question goes up to the large tier.
This chapter: How does a confidence threshold trade right answers against cost?
You will learn
- What a router and a fallback (a cascade) are
- How raising the threshold moves the quality and cost frontier
- Why a confident answer can still be wrong
Words you’ll meet (5)
- Router
- The step that decides which model answers a question.
- Cascade
- Ask the cheap model first; send the question to a bigger one only when the first answer looks unsure.
- Confidence
- A score from 0 to 1 for how sure an answer seems. Here it is a stand-in score set by hand to show the trade-off.
- Threshold
- The confidence below which the router sends a question up.
- Frontier
- The best right answers you can get at each cost: the line the router’s settings trace.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 3
The router asks the small tier first. Each small answer comes with a confidence score, and below the threshold the question goes up to the large tier, which answers it instead. Here the scores are stand-in numbers set by hand for this site, to show the trade-off; a real system computes them. Before you move the threshold:
FrugalGPT is an LLM cascade that learns which combinations of models to use for different queries; its authors report it can match the best single model (GPT-4) with up to 98% cost reduction. Source: arxiv.org (checked 2026-10-21)