Reading every page
no index257 pages read
Every page, to find the 6 that say it.
Module 26 · Search before AI
How a search box finds a page with no AI
Before chatbots, search was an index: a list of the pages each word is on. Here you build Larkspur’s, score its results and find where matching words stops being enough.
Read as
The whole idea in one example
“forklift”, two ways
Through the index
one drawer1 drawer
Its list names the same 6 pages, none read.
Pages are RAG’s 32-word pieces of Larkspur’s documents, searched as the page builds: a teaching search box, not any search engine’s numbers.
How it works
Four ideas carry the module
1 A drawer per word
An index lists the pages each word is on
Apache Lucene’s documentation: its terms index is an inverted index, because it can list, for a term, the documents that contain it.
Source, checked 25 October 2026
2 Rare words, fair repeats
BM25 weighs rarity, repeats and length
Robertson and Zaragoza (2009) call BM25 one of the most successful text-retrieval algorithms to come out of the probabilistic relevance framework.
Source, checked 25 October 2026
3 Words aren’t meaning
Cleaning helps; other words still miss
BEIR (Thakur et al., 2021) tested ten retrieval systems on 18 public datasets and found BM25 a robust baseline on data the systems were not trained for.
Source, checked 25 October 2026
4 Score it on test searches
Precision and recall over labelled searches
scikit-learn’s guide: precision is the share of what was picked that is right; recall is the share of the right ones that were found.
Source, checked 25 October 2026
The story
Larkspur’s search box, before the assistant
Before Larkspur Supply Co.’s AI assistant, staff found answers with a search box over the company’s documents; the team builds its index, tunes its scoring and finds where words stop working. You’re on the team that runs the search box.
- 257
- pages cut from Larkspur’s 47 documents
- 1,308
- drawers in the index, one per cleaned word
- 22
- test searches, each with the pages that answer it
- 32 of 35
- right pages in the top five for the best settings
Your path
The chapters
Chapter 2 · done
The index, word to pages
How does an index answer a search without reading the pages?
Who it’s for
- Engineers
- Product managers
- Analysts
- No maths needed
Anyone whose product has a search box, or who wants to know what AI search adds to it. The scoring formulas stay folded away; RAG picks up where this module ends.