Publish a web VIP with health checks on HAProxy

Tasks

1. Add the VIP address to LB1.
2. Write a frontend on the VIP and a backend with least connections.
3. Add an HTTP health check with sensible interval, rise and fall values, and slow start.
4. Pass the client IP to the servers.
5. Validate the config, reload, and prove failover by stopping one server.

Solution

1. 1Add the VIP on LB1LB1

   sudo ip addr add 10.20.0.100/24 dev ens192

   Make it permanent in netplan or NetworkManager. With a keepalived pair, the VIP floats between two load balancers instead, and net.ipv4.ip_nonlocal_bind=1 lets HAProxy bind to it on the standby node.

2. 2HAProxy configurationLB1

   # /etc/haproxy/haproxy.cfg
   global
       log /dev/log local0
       maxconn 20000
       stats socket /run/haproxy/admin.sock mode 660 level admin
   
   defaults
       mode http
       log global
       option httplog
       option forwardfor
       timeout connect 5s
       timeout client 30s
       timeout server 30s
   
   frontend fe_web
       bind 10.20.0.100:80
       default_backend be_web
   
   backend be_web
       balance leastconn
       option httpchk
       http-check send meth GET uri /healthz ver HTTP/1.1 hdr Host web.example.com
       http-check expect status 200
       default-server inter 2s fall 3 rise 2 slowstart 30s
       server web1 10.20.0.11:8080 check
       server web2 10.20.0.12:8080 check

   http-check send is HAProxy 2.2 and later. Older versions put the request on the option httpchk line. inter 2s fall 3 removes a dead server after about 6 seconds, rise 2 needs two good checks to return. option forwardfor adds X-Forwarded-For, because the servers see LB1's address as the source.

3. 3Validate and reloadLB1

   sudo haproxy -c -f /etc/haproxy/haproxy.cfg
   sudo systemctl reload haproxy

   Always check before reloading. A reload keeps existing connections; a restart drops them.

Verify

LB1# sudo haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid

LB1# echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18
# pxname,svname,status
fe_web,FRONTEND,OPEN
be_web,web1,UP
be_web,web2,UP
be_web,BACKEND,UP

PC1# for i in 1 2; do curl -s http://10.20.0.100/whoami; done
web1
web2

LB1# sudo journalctl -u haproxy -n 1 --no-pager -o cat
Server be_web/web2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Mistakes people make

• A health check that only opens TCP 8080. The app can return 500 on every request while the port stays open.
• A deep health check that queries the shared database. One database blip marks every server down at once.
• Forgetting the Host header in the check when the servers use name-based virtual hosts. The check gets a 404.
• Expecting the servers to log client IPs without reading X-Forwarded-For.
• Reloading without haproxy -c first.

On other platforms

create ltm monitor http mon_web_healthz defaults-from http interval 5 timeout 16 send "GET /healthz HTTP/1.1\r\nHost: web.example.com\r\nConnection: close\r\n\r\n" recv "200 OK"
create ltm pool pool_web load-balancing-mode least-connections-member monitor mon_web_healthz members add { 10.20.0.11:8080 10.20.0.12:8080 }
create ltm virtual vs_web destination 10.20.0.100:80 ip-protocol tcp profiles add { tcp http } pool pool_web source-address-translation { type automap }