How the vocabulary is learned

The sentence came out as more tokens than words: the tokenizer cut some words into pieces. The pieces are not chosen by hand. The tokenizer learned them from Larkspur’s own documents.

This chapter: Where does a tokenizer’s vocabulary come from, and why does it cut words where it does?

You will learn

A byte-level tokenizer starts from single bytes, so it can read any text
How a merge glues a common pair of tokens into one new token
Why the order of merges comes from the training text, not from the text you send

Words you’ll meet (3)

Byte: The smallest unit computers store text in. An English letter is one byte; many other characters take two to four.
Merge: A rule the tokenizer learned: wherever these two tokens sit side by side, glue them into one new token.
Byte pair encoding (BPE): Sennrich and colleagues (2016) brought byte pair encoding to neural translation: rare and unknown words are encoded as sequences of subword units.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 5

Here is the first paragraph of Larkspur’s company overview: “Larkspur Supply Co. sells office and warehouse hardware to businesses, together with the Larkspur Portal, a subscription ordering system. The company was founded in 2009 as a regional shelving distributor and now serves about 4,000 business customers.” A byte-level tokenizer starts from single bytes: its first 256 tokens are one per byte, so it can read any text in any language. Cut the paragraph into bytes.