Tinker Multimodal models

by Kondasamy Jayaraman · drawn with Colorfigures by April Zhu

...

A faxed invoice with a smudge on its total.

Chapter 6 of 7

When pictures mislead

Two images reach the assistant in the same week: a faxed invoice with a smudge on its total, and a returned crate whose label has a sentence written for the AI.

This chapter: How can an image make the assistant wrong, or make it do something it shouldn’t?

You will learn

  • Why a misread digit comes back with no sign of doubt
  • How a check outside the model catches it
  • Why text inside an image is data, never instructions
Words you’ll meet (3)
Misread
A reading that comes back confident and wrong, like a smudged 8 read as a 3.
Cross-check
A rule the host runs on the reading, such as “the lines must add up to the total”.
Prompt injection
Text written to steer a model, hidden in something it reads: an email, a web page, or an image.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 5

First, a supplier’s invoice that came by fax: grainy, with a smudge across the total. Read it with vision.