All fuses in
“Candidates receive an answer within ten …”
Module 20 · Interpretability
Why did the model say that?
Looking inside a model isn’t enough to explain it. Here you test its parts: pull a fuse, read its guess as it forms, copy an activation across, and push it off course.
Read as
The whole idea in one example
Pull one fuse and the guess changes
Layer 1, head 1 pulled
“Candidates receive an answer within ten …”
“business” drops out of the top 5. Larkspur mini’s numbers, computed from its real weights.
This prompt is one of the few where one head matters that much. Over all of Larkspur’s held-out text, the same fuse costs 74 of 659 right top-5 guesses: each head does a little, and the work is spread out. Finding that out, by experiment rather than by looking, is the module.
How it works
Four ways in
1 Looking isn’t proof
An attention map shows where a head looked
Jain and Wallace (2019) found that attention weights often disagree with other measures of what mattered, and that very different attention patterns can give the same prediction.
Source, checked 24 October 2026
2 Copy an activation across
Find where a word’s effect travels inside the run
Meng and colleagues (2022) used causal tracing, copying activations between runs, and found GPT's factual recall runs through middle-layer MLPs as it reads the subject's tokens.
Source, checked 24 October 2026
3 Read the guess forming
Read each block’s guess through the model’s last step
nostalgebraist's logit lens (2020) reads a model's middle layers through its own final output step, and finds a fair guess often forms partway up and is refined after.
Source, checked 24 October 2026
4 Push it
Add a direction and the model drifts
Turner and colleagues (2023) steer a model by taking the difference of its activations on two contrasting prompts and adding it during the forward pass, with no training.
Source, checked 24 October 2026
The story
Larkspur asks why
Larkspur Supply Co.’s team opens Larkspur mini again to answer a reviewer’s question: why did it say that? They pull its fuses, read its guess mid-way and push it. You’re on the team that has to answer the reviewer.
Played Transformers? This is the same Larkspur mini and the same engine room, now with a fuse box.
- 6
- fuses: one per head, one per MLP
- 11%
- of right top-5 guesses lost with Layer 1, head 1 pulled
- 199 of 256
- neurons fire on unrelated kinds of token
- 0
- weights changed by anything you do here
Your path
The chapters
Chapter 1 · done
Why did it say that?
What can an attention map tell you about why a model answered, and what can’t it?
Who it’s for
- Engineers
- Product managers
- Analysts
- No code needed
Engineers, product managers and analysts who hear “we looked inside the model” and want to know what that can and can’t show. It builds on Transformer architecture; any maths is optional.