TinkerLoading Larkspur’s toy experts model…
“We reply within five business”: every room works on each token.
Chapter 1 of 7
One big kitchen
A vendor’s model card says “47B total, 13B active”. Larkspur’s team wants to know what that means, starting with a plain model.
This chapter: In a plain model, how many weights work on each token?
About this chapter
You will learn
- Every weight of a dense model works on every token
- Why the work per token grows with size
- What a mixture of experts changes
Words you’ll meet
- Weight
- A number the model learned. A model’s size is how many weights it has.
- Dense model
- A model where every weight works on every token it reads.
- Mixture of experts
- A model split into many experts, of which only a few work on each token.
Step 1 of 4
Larkspur’s team starts with a : one big kitchen where every cook works on every token.
Predict first.