Smaller numbers

The server’s memory is mostly weights. Every weight is stored as a 16-bit number. Someone asks whether 8 bits, or even 4, would do.

This chapter: What does storing the weights in fewer bits save, and what does it cost?

You will learn

Quantization: rounding each weight to one of a few levels
How much memory 8 and 4 bits save, counted from the model’s shape
How to check what it costs: held-out text and the top 5 next tokens

Words you’ll meet (4)

Quantization: Storing numbers with fewer bits by rounding each to the nearest of a fixed set of levels.
int8, int4: Whole numbers stored in 8 or 4 bits: 255 or 15 levels, scaled to each row of weights.
Held-out text: Text kept out of training, used only to score the model.
Loss: How surprised the model is by the next piece of text, on average: lower is better.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 5

Larkspur mini’s weights are 16-bit numbers. Quantization rounds each one to a few levels: here each row of weights gets its own scale, its largest weight maps to the top level, and every other weight to the nearest. Store them in 8 bits.

Loading Larkspur mini