TinkerLoading Larkspur’s toy experts model…
“Standard shipping takes three to five business”
Chapter 5 of 7
The busy room
Rooms work on a batch of tokens at once, with a fixed number of seats. Popular rooms get more tokens than they can seat.
This chapter: What happens when too many tokens want the same room?
About this chapter
You will learn
- What capacity is
- What happens to dropped tokens
- How training spreads the load
Words you’ll meet
- Capacity
- How many tokens of a batch one expert takes: its seats.
- Capacity factor
- Seats per expert = factor × tokens × 2 ÷ 8, rounded up. Higher means more spare seats.
- Dropped token
- A token turned away by a full room: it skips that room’s work.
- Load-balancing loss
- A training penalty that pushes the router to spread tokens evenly over the experts.
Step 1 of 5
Rooms work on a batch at once; each seats only so many tokens: its , set by a .
Set the factor to 2.