Nothing on
obeyedIt followed the notice and paid $510.00 on bw-billing.example.
Module 31 · Computer use
How does an agent use a website made for people?
It looks at a screenshot and answers with a point to click. Here you watch an agent re-order stock on a supplier’s old portal, predict its clicks, and find the screens that fool it.
Read as
The whole idea in one example
A notice on the page, addressed to the agent
Allow-list on
stoppedIt still tried. The host refused the page, and the re-order went on.
The agent’s actions were written for this site to show each lesson; they were not recorded from a real model. The screens are drawn, and the host’s rules run live.
How it works
Six ideas carry the module
1 Look, then click
A screenshot in, a point out, again and again
Repeating the request, the action and the returned screenshot without user input is what Anthropic calls the agent loop.
Source, checked 25 October 2026
2 The wrong button
A resized screenshot’s point must be scaled back
Coordinates are in screenshot pixels: if you scale screenshots down before sending them, scale Claude’s coordinates back up before clicking on the real display.
Source, checked 25 October 2026
3 Too quick
A screenshot taken mid-load shows the past
Some applications need time to respond to actions; Anthropic’s example pauses after each click so the screen can update, and the tool has a wait action for pages still loading.
Source, checked 25 October 2026
4 The page that gives orders
Words on screen can steer the agent
Claude sometimes follows commands found in content even when they conflict with your instructions: instructions on web pages or in images might override yours.
Source, checked 25 October 2026
5 Ask before you pay
A person approves the clicks that spend money
Anthropic advises asking a person to confirm decisions with real-world consequences and anything needing consent, such as completing a payment or agreeing to terms of service.
Source, checked 25 October 2026
6 Every look costs
Smaller screenshots cost less, until the agent misses
Claude reads images in 28 × 28-pixel patches, one visual token each, so a screenshot costs ⌈width / 28⌉ × ⌈height / 28⌉ tokens.
Source, checked 25 October 2026
Steel’s OSWorld leaderboard, last updated 11 October 2026, lists several agents above the paper’s 72.36% human baseline, the highest at 86.1%; third-party boards disagree on the leader. Source, checked 25 October 2026.
The story
Larkspur’s re-orders, on a portal with no API
Larkspur Supply Co. re-orders stock from Brightwater Fasteners, whose old trade portal has no API, so an agent clicks through it like a person. You set the rules the agent’s host follows.
AI agents showed the loop with tools; Multimodal models showed how a picture becomes tokens. Here the tool is the screen. The workbench’s best policy is one of 1.
- 5
- screenshots for one re-order
- 1,334
- tokens a 1280 × 800 screenshot costs
- 72.36%
- of OSWorld’s computer tasks people completed
- 52%
- fewer tokens with the workbench’s best policy
Your path
The chapters
Chapter 7 · done
Workbench: a policy for the agent
Which host rules and screenshot size get every re-order right and stop every attack?
Who it’s for
- Engineers
- Product managers
- Analysts
- No code needed
Product managers, engineers and analysts weighing an agent that works through screens. AI agents and Multimodal models come first; no code or maths needed.