TinkerLoading Larkspur’s toy experts model…
“Every enterprise account has a named customer success”
Chapter 6 of 7
Total versus active
Back to the vendor’s card: “47B total, 13B active”. Larkspur needs to know what each of the two numbers costs.
This chapter: What does the total cost, and what does the active count cost?
About this chapter
You will learn
- Total weights set the memory
- Active weights set the work per token
- What the toy measured against a dense model
Words you’ll meet
- Total parameters
- Every weight in the model, every expert included. All of them must sit in memory.
- Active parameters
- The weights one token runs through: shared parts plus its chosen experts.
- Memory
- The GPU memory that holds the weights while the model serves.
Step 1 of 5
Every room’s weights wait in , ready, but each token works through only two rooms a floor.
Count the toy’s weights.