Loading Larkspur mini

Layer 1 · head 1 · token 7 of 7 (␣business)Each desk is one head. Weights computed live from Larkspur mini.

Chapter 4 of 7

Heads and layers

One attention pattern can’t look for everything at once. Larkspur mini has several, stacked in layers, and nobody told any of them what to look for.

This chapter: Why does a model have many heads and many layers, and how do their results come together?

You will learn

That each head learns to look for something different
How to spot a head that finds the start of the current word
How every layer adds to one running vector per token: the residual stream

Words you’ll meet (3)

Head: One attention pattern with its own learned weights. Multi-head attention runs several heads side by side, so the model can attend to different kinds of information at once.
Layer: One round of attention heads followed by a small calculation on each token. Layers are stacked: the second reads what the first wrote.
Residual stream: The running vector each token carries through the model. Every layer adds what it found to it instead of replacing it.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 6

One attention pattern can only look for one kind of thing. So each layer runs several side by side, called heads, each with its own weights. Larkspur mini has 2 layers with 2 heads each: the desks in the engine room. Light every head on the same prompt and watch the threads change.

Loading Larkspur mini…