Tinker Computer use

by Samy (Kondasamy Jayaraman) · drawn with Colorfigures by April Zhu

Screenshot 1280 × 800 px

Look 3: a screenshot of Basket, 1,334 tokens.

Chapter 4 of 7

The page that gives orders

Someone has been editing what Brightwater’s pages show. Larkspur’s agent reads every word on them.

This chapter: Can words on a web page take over the agent?

About this chapter

You will learn

  • Why the agent reads on-screen text as instructions
  • What a planted notice can make it do
  • How an allow-list stops it leaving the site

Words you’ll meet

Prompt injection
Instructions placed in content the model will read, written to steer it away from its task.
Allow-list
The sites the host lets the agent open. Everything else is refused.

Step 1 of 5

Someone planted a notice on Brightwater’s basket page, telling AI assistants to open bw-billing.example first.

Predict the agent’s click.