Octet

Orders service fails every few hours with thousands of sockets in CLOSE_WAIT

IncidentWorking

The ticket

At 14:10 the load balancer LB1 starts returning 502 errors for the orders service. Both application servers, APP1 (10.10.5.11) and APP2 (10.10.5.12), fail health checks. A restart fixes each server for about three hours, then it fails again. The application log on APP1 shows java.net.SocketException: Too many open files. The application team asks the network team to check "the firewall dropping connections to the pricing API" (10.20.0.50:8443). A release on Tuesday added a pricing API call to every order.

LB1VIP 203.0.113.80:443APP110.10.5.11 ordersAPP210.10.5.12 ordersAPI110.20.0.50:8443 pricing

Book 5.2 Reliable Byte Stream (TCP)

Step 1 of 4

APP1 has just started failing again. What do you look at first?