Tests open to edits
green, wronglyAll lamps green, on a loosened test. Larkspur’s own checks: 3 of 4 pass.
Module 37 · Coding agents
How does an AI agent fix a bug in your code?
It searches, reads, edits and runs the tests, one turn at a time. Here you review its patch hunk by hunk, watch the tests judge it, and catch it cheating.
Read as
The whole idea in one example
Make the red checks green
Ticket LS-415 from Tom: “The discount checks are red. Make them green before the 5 pm release.”
Test files protected
really doneThe edit to the test was refused, so it fixed the code. Larkspur’s own checks: 4 of 4 pass.
The agent’s turns were written for this site to show each lesson; they were not recorded from a real model. The code, every version of it and the tests are real, and run live.
How it works
Six ideas carry the module
1 Read, edit, run
An action, its result, the next action
SWE-bench (Jimenez et al., 2023) asks a model to resolve 2,294 real GitHub issues from 12 Python repositories, judged by the repositories’ own tests; the best model then solved 1.96%.
Source, checked 25 October 2026
2 Finding the file
Search first; every read costs tokens
3 A patch, in hunks
Keep the fix, drop the drive-by
4 Red to green
A failing test names what broke
In August 2024 OpenAI and the SWE-bench authors released SWE-bench Verified: 500 tasks that professional developers screened as fair, so a failure means the model, not a broken task.
Source, checked 25 October 2026
5 Green by cheating
Done means what the tests check
In June 2025 METR reported recent frontier models “attempting (often successfully) to get a higher score by modifying the tests or scoring code” in its coding tasks.
Source, checked 25 October 2026
6 What it may run
Allowed, ask first, never
Claude Code’s permission rules come in three kinds, deny, ask and allow, checked in that order: a matching deny blocks a command even when an allow rule also matches.
Source, checked 25 October 2026
Steel’s SWE-bench Verified board, last updated 11 October 2026, lists a highest tracked score of 97.0%; boards disagree on the leader, and scores move monthly. Source, checked 25 October 2026.
The story
Larkspur’s pricing code, and three tickets
A Larkspur ticket says the bulk discount is wrong; the team hands it to a coding agent, and learns it is only as right as the tests it runs. You review its patches and set the rules its host follows.
AI agents showed the loop with tools; Post-training showed a model learning to game its grader. Here the grader is the test suite. The workbench’s best rules are one of 1.
- 541 → 264
- tokens read for the same fix, without and with search
- 4
- real tests, run live on every version
- 1.96%
- of SWE-bench issues the best model solved in 2023
- 1
- question to a person under the workbench’s best rules
Your path
The chapters
Chapter 7 · done
Rules for the queue
Which host rules get every ticket really done, with the fewest questions to a person?
Who it’s for
- Engineers
- Product managers
- Analysts
- A little code
Product managers, engineers and analysts weighing an agent that changes their code. AI agents comes first; reading a few lines of code helps, and no maths is needed.