TinkerThe map appears once a question reaches the index.
RAG’s map of meaning: the diamond is “What is the hotel nightly limit for business travel?”.
Chapter 5 of 7
From token to vector
Tokens are numbers, but an id says nothing about meaning: id 500 is no closer to id 501 than to id 9. Larkspur’s search, built in the RAG module, needs text that means the same to sit close together.
This chapter: How does text become a list of numbers that carries its meaning?
You will learn
- What an embedding is: a list of numbers for a piece of text
- Why texts with similar meaning get similar embeddings
- How RAG’s map of meaning draws them in two dimensions
Words you’ll meet (3)
- Embedding
- OpenAI: an embedding is a list of numbers, and the distance between two embeddings measures how related their texts are.
- Vector
- A list of numbers; an embedding is one.
- Dimension
- One number in a vector. RAG’s embedding model gives 384.
The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.
Step 1 of 4
A token id is just a row number in the tokenizer’s list: it says nothing about meaning. For search, Larkspur needs numbers where similar meanings sit close together. An embedding model gives them: it reads a piece of text and returns a list of numbers, a vector. RAG’s model gives 384 numbers per text, and the RAG module already embedded every piece of Larkspur’s documents and every customer question. Before you compare three questions: