Orders service fails every few hours with thousands of sockets in CLOSE_WAIT
IncidentWorking
The ticket
At 14:10 the load balancer LB1 starts returning 502 errors for the orders service. Both application
servers, APP1 (10.10.5.11) and APP2 (10.10.5.12), fail health checks. A restart fixes each server for
about three hours, then it fails again. The application log on APP1 shows
java.net.SocketException: Too many open files. The application team asks the network team to check
"the firewall dropping connections to the pricing API" (10.20.0.50:8443). A release on Tuesday added a
pricing API call to every order.
Step 1 of 4
APP1 has just started failing again. What do you look at first?