Round robin
p991,000 ms
Birch still gets every fourth request.
Module 35 · Load balancing
How many servers share one shop’s crowd
One server has a ceiling; several need something to pick one for every request. Here you put Larkspur’s shop behind a load balancer and keep a sale fast through a crash, a slow server and full carts.
Read as
The whole idea in one example
One slow server, two rules
Least outstanding
p99288 ms
Birch’s requests pile up, so it gets fewer.
Waits come from a simulated sale: Larkspur’s own traffic and planning numbers, not a measurement of any real server or balancer.
How it works
Four ideas carry the module
1 Take turns
Round robin sends each server the next request
NGINX sends requests to its servers in round-robin fashion unless another method is set: round robin is its default.
Source, checked 25 October 2026
2 The least busy
Counting waiting requests steers around a slow server
NGINX’s least-connected method sends the next request to the server with the fewest active connections, which spreads load more fairly when some requests take longer.
Source, checked 25 October 2026
3 Is it alive?
A health check pulls a dead server
An AWS Application Load Balancer checks each target every 30 seconds by default and takes it out of service after 2 failed checks in a row, its default unhealthy threshold.
Source, checked 25 October 2026
4 The slow tail
A hedged copy rescues the slowest requests
Dean and Barroso describe hedged requests: send the same request to a second server once the first has waited longer than most requests do, and use whichever answers first.
Source, checked 25 October 2026
The story
Larkspur’s shop outgrows one server
Larkspur Supply Co.’s shop outgrows one server; the team puts four behind a load balancer and keeps a sale fast through a crash, a slow server and full carts. You’re on the team that runs the shop.
Played Inference engineering? It shows one server from the inside; this module spreads the crowd over several.
- 395
- requests in Larkspur’s sale, the traffic every chapter runs
- 4
- servers behind the balancer, one twice the size of the others
- 465 ms
- p99 with Ash alone in the sale
- 190 ms
- p99 of the best setup in chapter 7, with one server dead and one slow
Your path
The chapters
Chapter 1 · done
One server’s ceiling
What happens to the wait when requests arrive faster than one server can answer?
Chapter 5 · done
Same customer, same server
How do you keep a customer’s cart, and what happens when its server dies?
Chapter 6 · done
The slow tail
Why does one slow server set the slowest customers’ wait, and how do you cut it?
Chapter 7 · done
Route the sale
Which rule, health check and hedge keep the slowest customers fastest?
Who it’s for
- Engineers
- Product managers
- Analysts
- No code needed
Anyone whose product runs on more than one server, and who wants to know why a dashboard can say every server is healthy while customers wait. Knowing what a server and a request are is enough.