The latency budget

The phone line works, and the first callers say the same thing: the pause after they stop talking feels long. The team times every stage.

This chapter: Where does the wait go, and how do you bring the first word forward?

You will learn

Why the stages’ times add up into the wait the caller hears
What streaming changes: the voice starts on the first sentence while the model writes the rest
Why the first sound matters more than when the reply is finished

Words you’ll meet (3)

Latency: How long something takes to respond. On a call, the gap between the caller’s last word and the first sound of the reply.
Streaming: Passing work on in pieces as it is made, instead of waiting for the whole thing.
Latency budget: The wait you can afford, shared out between the stages that make it up.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 4

Latency is how long something takes to respond; on a call, the gap between the caller’s last word and the first sound of the reply. Time Dana’s call from chapter 1, stage by stage.