Tokens are not characters

The mask works. But a schema is written in characters, and the model writes tokens: pieces of one to several characters, as you saw in Tokens. The mask has to judge whole tokens.

This chapter: How does a mask written for characters judge a token that carries several?

You will learn

Why the mask checks every byte of a token, not just its first
How a length cap blocks long tokens before short ones
What token healing fixes at the boundary between prompt and answer

Words you’ll meet (3)

Byte: The smallest unit of text a computer stores. Larkspur mini’s tokens are runs of bytes; in plain English text one byte is one character.
Length cap: The most characters a free-text field may hold. Here every free field has one, so the answer always closes.
Token healing: Backing up over the prompt’s last token and letting the model’s first token start with it, so a prompt that ends mid-token is read the way the training text was split.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 4

A new schema: one field, a free note, capped at 8 characters so it always closes inside Larkspur mini’s 32-token window. That cap, maxLength, is this site’s addition to the schema subset: Anthropic’s structured outputs reject it. The model writes tokens, not characters: a token here can carry one character or several, as you saw in Tokens. Before you look near the cap:

The schema gives the note at most 8 characters. The model has written some, and two characters of room are left. Which tokens can the mask still allow?