The reply
wrong“What time do I have to order by for same-day shipping?”
Order before 12 noon Central time and it ships the same day.
Quarry Lane Builders got last year’s answer.
Module 14 · Observability
Read what the assistant really did, and fix what matters first
Offline evals test the questions you thought of. Production shows the ones you didn’t. Here you read a day of Larkspur’s traces, note what went wrong, group the notes into failure modes, and turn the worst into tests.
Read as
The whole idea in one example
The reply is where you see it; the trace shows where it began
The first failure
retrieval policiesTab to the trace, then use the arrow keys to read each span.
Found: Shipping guide (2024): same-day cutoff 12 noon Central
Note the first failure you see in a trace: upstream errors can cause the problems that show up downstream. These traces are written by hand for this module, shaped like real ones, so they show the method, not any system’s failure rates. Source, checked 21 October 2026.
How it works
Four moves carry the module
1 Read the trace
Every step of one request, nested and timed
OpenTelemetry keeps semantic conventions for generative AI: shared names for the spans, metrics and events of model calls, agents and tools. The span conventions are still marked Development.
Source, checked 21 October 2026
2 Note what went wrong
A free note per failed trace, at its first failure
Open coding: a reviewer reads traces and writes open-ended notes on any issue they see, like a journal. The method is adapted from qualitative research.
Source, checked 21 October 2026
3 Group and count
Notes that share a cause become a failure mode
Axial coding sorts the open notes into a failure taxonomy, then counts the failures in each category. Hamel Husain calls it the most important step.
Source, checked 21 October 2026
4 Fix what matters
How often, times how much harm
Hamel Husain calls error analysis the most important activity in evals: it finds the failure modes unique to your application and data, and tells you which evals to write.
Source, checked 21 October 2026
The story
Larkspur’s assistant meets real customers
Larkspur Supply Co.’s assistant is answering real customers now, so the team reads its traces, names what goes wrong, and fixes what matters first. You’re the person on the team who reads the traces: you know what a right answer looks like.
In Evals the team graded a suite written before launch. This module starts from production traffic instead. The most frequent failure here, “search missed the passage”, is not the one to fix first: chapter 5 shows why.
- 40
- traces in Larkspur’s day, written by hand for this module
- 18
- of them failed, in 5 modes
- 30
- traces to note yourself before you trust an AI’s suggestions
- 100
- traces or so in a working review pool
Your path
The chapters
Chapter 1 · done
What the assistant did
What happened, step by step, when a customer asked the assistant a question?
Chapter 3 · done
Open coding
How do you start making sense of failures before you know what kinds there are?
Who it’s for
- Engineers
- Product managers
- Analysts
- No code needed
Product managers, engineers and analysts whose assistant is live, or about to be. Evals comes first; no code or maths needed. There are 5 failure modes to find.