How do load balancer health checks work, and how do you design a good one?
Show the answer
Say this first
The load balancer probes each pool member at an interval, marks it down after a number of consecutive failures and up again after a number of successes. Probes range from ICMP and TCP connect to an HTTP request that must return a given status or text. A good check tests what that server needs to serve requests, through a lightweight endpoint like /healthz, without depending on a shared backend that could mark every member down at once, and uses rise and fall counts to avoid flapping.
Go deeper
Probe types
| Probe | Proves | Misses |
|---|---|---|
| ICMP | Host is up | The service |
| TCP connect | Port is listening | The app returning errors |
| HTTP with expected status or string | The app answers correctly | Deep dependency problems, by design |
| Passive (real traffic) | Real users get errors | Idle servers |
Timers you should know
- HAProxy defaults:
inter 2s,fall 3,rise 2. A dead server is removed after about 6 seconds. - F5 HTTP monitor defaults: interval 5 seconds, timeout 16 seconds.
Design rules
- Shallow, not deep. If every server's check queries the same database, a database hiccup removes the whole pool and turns a slowdown into an outage.
- Drain gracefully. Fail the check or set the member to drain before maintenance so existing sessions finish.
- Know the all-down behavior. Some balancers return 503, others fail open. AWS ALB, for example, sends traffic to all targets when every target is unhealthy.
- Probe source. Checks come from the load balancer's own address. Firewalls and host ACLs must allow it, and a green check does not prove that client traffic works.
They will ask next
How do liveness and readiness checks in Kubernetes map to this?
Readiness is the load balancer's question: should this pod get traffic. A failed readiness probe removes the pod from the Service endpoints. Liveness asks whether to restart the container.
Traps
- Checking only that TCP 443 is open while the application behind it returns 500.
How did you do?