How do target health checks and cross-zone load balancing work on AWS load balancers?
Show the answer
Say this first
Each target group has its own health check. For an ALB the defaults are HTTP on the traffic port, path
/, every 30 seconds, 5 second timeout, healthy after 5 passes and unhealthy after 2 failures. Removed
targets drain for the deregistration delay, 300 seconds by default. Cross-zone load balancing spreads
traffic over targets in all AZs: it is always on at the ALB level and off by default on an NLB.
Go deeper
Health checks
- Probes come from the load balancer nodes' private IPs. The target security group must allow them, ideally by referencing the load balancer's security group.
- Point the check at a real health endpoint (
/healthz) that proves the instance itself can serve, not just that the web server answers. Leave shared dependencies such as the database out of it, or one database blip marks every target unhealthy at once. - Success codes default to
200for an ALB; you can widen it, for example200-399. - Fail open: if every target in a target group is unhealthy, the load balancer sends traffic to all of them anyway. A bad health check path does not cause an outage, but it hides the real health.
Deregistration delay
During scale-in or deploys, a target stops getting new connections but keeps existing ones for up to 300 seconds. Short-lived HTTP services often lower it to 30 seconds to speed up deploys.
Cross-zone
| ALB | NLB | GWLB | |
|---|---|---|---|
| Default | On (can be turned off per target group) | Off | Off |
| Cost when on | No inter-AZ charge | Inter-AZ data charges apply | Inter-AZ data charges apply |
With cross-zone off, each AZ's load balancer node only uses targets in its own AZ. If AZ a has 2 targets and AZ b has 8, each AZ still gets half the traffic, so the 2 targets in AZ a run four times hotter.
They will ask next
All targets show unhealthy but the site still works. How?
Fail-open behaviour. The load balancer routes to all targets when none are healthy. Usually the health check path or port is wrong, or the security group blocks the probes.
What is slow start mode?
An ALB option that ramps a newly healthy target up to its full share over a set period, so a cold cache or JIT does not get the full load at once.
Traps
- Forgetting to allow the health check from the load balancer's security group.
- Uneven target counts per AZ with cross-zone off.