What is a PMTUD black hole, and how do you find and fix one?
Show the answer
Say this first
Path MTU discovery relies on routers sending ICMP "fragmentation needed" (IPv4 type 3 code 4) or "packet too big" (ICMPv6 type 2) back to the sender when a packet is too large to forward without fragmenting. If a firewall drops that ICMP, or the router does not send it, the sender never learns, keeps sending full-size packets, and they vanish. Connections hang after the handshake. I find it by bisecting with DF pings and fix it by permitting that ICMP, clamping MSS, or setting the right MTU where the tunnel starts.
Go deeper
How PMTUD works
- TCP stacks send segments with DF set.
- A router whose next link is too small drops the packet and returns ICMP type 3 code 4, including the next-hop MTU (RFC 1191).
- The sender lowers its path MTU for that destination and resends smaller segments.
IPv6 routers never fragment. Every too-big packet is dropped with ICMPv6 type 2 (RFC 8201), so filtering that message breaks IPv6 paths outright.
Ways it breaks
- A firewall blocks all ICMP "for security".
- The router that drops the packet has
no ip unreachableson that interface, or rate-limits ICMP hard. - The ICMP comes from a private address inside a provider, and the sender's edge filters it.
- ECMP or an anycast load balancer delivers the ICMP to a different server than the one holding the connection.
Finding it
- DF ping bisect from the client side.
- A capture at the sender shows full-size segments retransmitted with no ICMP arriving. A capture at the tunnel router shows whether the ICMP is generated and leaves.
Fixing it
- Permit ICMP type 3 code 4 and ICMPv6 type 2 everywhere. Rate-limit if you must; never block.
- Clamp TCP MSS at the tunnel:
ip tcp adjust-mss 1360. - Hosts can use packetization layer PMTUD (RFC 4821), which probes using TCP itself. On Linux,
net.ipv4.tcp_mtu_probing=1turns it on when a black hole is detected.
They will ask next
Why not just clear the DF bit on the router?
Then the router fragments, which works but costs CPU on the router and the receiver, and one lost fragment loses the whole packet. It is a workaround for a broken path, and it does nothing for IPv6.
Does MSS clamping help UDP?
No. MSS is a TCP option. Large UDP, such as some VPN protocols or big DNS answers, still depends on a correct MTU or fragmentation.
Traps
- Blocking all ICMP as a security measure. It breaks PMTUD, and IPv6 cannot work properly without ICMPv6.