Loading Larkspur mini

Layer 1 · head 1 · token 7 of 7 (␣business)Who looks at whom, in Larkspur mini’s first head. Weights computed live.

Chapter 3 of 7

Attention

Each token now enters the model as its own vector. But “business” alone can’t tell the model what comes next: it needs “five” and “within” too. Something has to let tokens read each other.

This chapter: How does each token gather what it needs from the tokens before it?

You will learn

How to read an attention heatmap: who looks at whom, and how much
Why each row of weights adds up to 1
Why a token can never look at a token after it: the causal mask

Words you’ll meet (3)

Attention: The step where each token gives every earlier token a weight, then takes in a mix of them, more from the heavier ones.
Weight: How much one token takes from another, between 0 and 1. A token’s weights add up to 1.
Causal mask: The original Transformer's decoder masks attention so a position cannot look at the positions after it, which keeps generation one token at a time.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 5

Each token is now a vector. To guess what follows “business”, the model has to know “five” and “within” came before it. Attention is how tokens read each other: each token gives every token a weight, then takes in a mix of their vectors, more from the heavier ones. Open the Attention tab above the picture to see the weights.

Loading Larkspur mini…