TinkerLook 1: a screenshot of Home, 851 tokens.
Chapter 2 of 7
The wrong button
To save tokens, the team now sends the agent smaller screenshots. The real screen hasn’t changed.
This chapter: Why would an agent that aims well click the wrong button?
About this chapter
You will learn
- Why a click comes back in the screenshot’s pixels
- How the host scales it back to the screen
- What goes wrong when it forgets
Words you’ll meet
- Coordinates
- A point as x (pixels from the left) and y (pixels from the top).
- Scaling back
- Turning a point in a resized screenshot into the same point on the real screen, by dividing by the resize factor.
Step 1 of 4
The agent now gets 1024-pixel screenshots of a 1280-pixel screen. This host clicks the agent’s point unchanged.
Click where the click lands.