TinkerLoading Larkspur’s toy experts model…
“We reply within five business”
Chapter 2 of 7
Rooms and a doorman
The vendor’s model splits its big kitchen into rooms. A doorman at each floor sends every token to a few of them.
This chapter: Who picks the rooms for a token, and how many?
About this chapter
You will learn
- What an expert is
- How the router scores every expert
- Why only the top two run
Words you’ll meet
- Expert
- One of the rooms: a small MLP of its own. Each layer here has eight.
- Router
- The doorman: a small layer that scores every expert for each token.
- Top-2
- Only the two best-scored experts run; their outputs mix by the router’s scores.
Step 1 of 5
Each floor holds 8 rooms, the . Its doorman, the , scores every room for each token.
Predict first.