Why do load balancers use SNAT, and how does it compare with routed mode and direct server return?
Show the answer
Say this first
In one-arm mode the load balancer changes the destination from the VIP to a server. If it keeps the client's source IP, the server replies straight to the client through its default gateway, the client sees a reply from an address it never contacted, and the connection fails. SNAT rewrites the source to a load balancer address so replies come back through it. The costs are losing the client IP (restored with X-Forwarded-For or PROXY protocol) and SNAT port limits. Routed mode makes the load balancer the servers' gateway and keeps client IPs. Direct server return sends only requests through the load balancer and servers reply from the VIP directly, which is fast but layer 4 only.
Go deeper
| Mode | Return path | Client IP at the server | Notes |
|---|---|---|---|
| SNAT, one-arm | Back through the load balancer | Lost, restore with XFF or PROXY | Easiest to insert into an existing network |
| Routed, inline | Through the load balancer as gateway | Preserved | All server traffic crosses it |
| Direct server return | Server to client directly | Preserved | Layer 4 only, servers need the VIP on loopback |
SNAT port exhaustion
Every connection to the same server IP and port needs a unique SNAT address and port pair, so one SNAT address supports roughly 64,000 concurrent connections per server and port. Busy VIPs use a SNAT pool of several addresses. The symptom is intermittent connection failures under load.
Direct server return on Linux servers
ip addr add 10.20.0.100/32 dev lo
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2The VIP on loopback lets the server accept packets for it. The two sysctls stop the server answering ARP for the VIP so only the load balancer owns it on the segment. The load balancer rewrites only the destination MAC, so servers must share the layer 2 segment, unless you use IP-in-IP tunneling as Linux IPVS tunnel mode does.
Restoring the client IP with SNAT
- HTTP: X-Forwarded-For (HAProxy
option forwardfor, an F5 HTTP profile setting). - Any TCP: PROXY protocol prepended to the connection. The server must be configured to expect it.
They will ask next
Why don't health checks catch a missing SNAT?
Probes are sourced from the load balancer's own address, so their replies come back to it. Only client traffic, sourced from client addresses, takes the broken return path.
Traps
- Turning on PROXY protocol at the load balancer but not on the server. The server sees junk at the start of every connection.