Tinker Multimodal models

by Kondasamy Jayaraman · drawn with Colorfigures by April Zhu

...

Packing slip, not read yet.

Chapter 3 of 7

Reading a scan

Not every page is a clean invoice. The warehouse sends a packing slip with a table, and a handwritten note stuck to a delivery. There is an older way to read a scan, too: turn it into text first.

This chapter: When is reading the image better than reading its text?

You will learn

  • What OCR is, and what it hands the model
  • Why OCR breaks tables and handwriting
  • Why a vision model keeps a table’s rows and columns together
Words you’ll meet (3)
OCR
Optical character recognition: software that turns an image of text into plain text. The model then reads only the text.
Vision
Here, sending the image itself: the model reads its patches and sees the layout.
Layout
Where things sit on the page: rows, columns, which number belongs to which line.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 3

There are two ways to read a scan. OCR (optical character recognition) turns the image into plain text, and the model reads only the text. Vision sends the image itself, and the model reads its patches. Before you try both:

The packing slip is a table: SKU, item, ordered, shipped. Which reading keeps each quantity with its own line?