Tinker Multimodal models

by Kondasamy Jayaraman · drawn with Colorfigures by April Zhu

...

The invoice at 1,600 px, not read yet.

Chapter 4 of 7

Pulling fields out

Accounts wants each invoice’s number, date, order, total and terms typed into the books. The assistant can pull them out, but someone has to know whether it got them right.

This chapter: How do you check an extraction field by field?

You will learn

  • What extracting fields from a document means
  • Why grading each field finds what one overall score hides
  • How OCR and vision compare on cost for clean print
Words you’ll meet (3)
Extraction
Pulling named fields (a total, a date) out of a document into a fixed form.
Field
One named value the form needs, like the invoice number.
Reference
The right value for each field, written by a person, that the extraction is graded against.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 4

Extraction means pulling named fields out of a document into a fixed form: here the invoice’s number, date, order number, total and payment terms. Each field is graded against a reference, the right value written by a person, with the Evals module’s key-fact check. Before you run it:

At 1,600 px, OCR hands the model 195 text tokens and vision 2,610 image tokens. Which gets more of the invoice’s five fields right?