Why does TIME_WAIT exist, how long does it last, and is a server with 30,000 sockets in TIME_WAIT a problem?
Show the answer
Say this first
The side that closes first waits two maximum segment lifetimes for two reasons: to re-send the last ACK if the peer's FIN is retransmitted, and to let stray segments from this connection expire before the same four-tuple can be reused. RFC 793 set MSL at 2 minutes, so 4 minutes; Linux uses a fixed 60 seconds. Thirty thousand TIME_WAIT sockets on a server cost little memory and are normal. The real risk is on a client or proxy opening many short connections to one destination IP and port, where TIME_WAIT can use up the ephemeral ports.
Go deeper
Why it is cheap on a server
A server's TIME_WAIT entries all share its listening port and differ by client address and port, so they never block new connections. Linux keeps them in a small structure without buffers.
Where it hurts
A proxy, load balancer or application server connecting to one backend 10.0.0.50:8080 needs a unique
source port per connection. Linux's default range is 32768 to 60999, so 28,232 ports. If each
connection is closed by the client and waits 60 seconds, the ceiling is about 28,232 / 60, roughly 470
new connections per second to that one backend address and port. Past that, connect() fails with
EADDRNOTAVAIL ("Cannot assign requested address").
Fixes, best first
- Reuse connections: HTTP keep-alive, connection pools, gRPC.
- Let the server close first, so TIME_WAIT lands on the side that is not short of ports.
- Add source IPs or backend IPs, since the limit is per four-tuple.
- Widen
net.ipv4.ip_local_port_range. net.ipv4.tcp_tw_reuse=1lets new outgoing connections reuse a TIME_WAIT four-tuple when timestamps show it is safe. Recent kernels default to 2, which means loopback only.
What not to do
tcp_tw_recycle was removed in Linux 4.12. It dropped SYNs from clients behind NAT whose timestamps did
not increase across hosts. Closing with SO_LINGER set to 0 avoids TIME_WAIT by sending RST, and throws
away unsent data.
They will ask next
Does net.ipv4.tcp_fin_timeout shorten TIME_WAIT on Linux?
No. Despite the name, it limits how long an orphaned socket stays in FIN_WAIT_2. TIME_WAIT is a compile-time 60 seconds on Linux.
How do you see TIME_WAIT sockets and their remaining time?
ss -tano state time-waitlists them with a timer such astimer:(timewait,41sec,0). Usess -sfor a summary count.Why is 2 x MSL enough to protect a new connection?
MSL is the assumed longest time a segment can survive in the network. Waiting twice that covers a segment travelling one way plus a reply travelling back, so nothing from the old connection can still be in flight when the four-tuple is reused.
Traps
- Tuning servers to "fix" TIME_WAIT counts that cause no errors. Measure port exhaustion first.
- Recommending tcp_tw_recycle. It no longer exists and broke clients behind NAT.