Tinker Diffusion language models

by Samy (Kondasamy Jayaraman) · drawn with Colorfigures by April Zhu

“ safety boots and high-visibility” and 16 blanks.

Chapter 3 of 7

Sure words first

One pass guesses every blank. The decoder must choose which guesses to keep; the rest wait for the next pass.

This chapter: Which blanks should a pass stamp first?

About this chapter

You will learn

  • What remasking chooses on each pass
  • Why keeping the surest guesses works better than random
  • Why the frame fills in any order

Words you’ll meet

Remasking
Choosing which guesses a pass keeps; the rest go back to blanks for the next pass.
Surest first
Keep the guesses with the highest confidence. Papers call it low-confidence remasking.
Careful fill
One slot a pass, surest first: the slowest way the toy writes, used here as the yardstick.

Step 1 of 4

Each pass stamps 4 of the 16 blanks, so the reply takes 4 passes. decides which 4 to keep.

Predict first.

Each pass may keep 4 of the guesses. Which 4 should it keep?