Hosts can ping each other across a VXLAN fabric, but large file transfers hang. What do you suspect and how do you prove it?
Show the answer
Say this first
MTU. A host packet of 1500 bytes becomes a 1550-byte outer IP packet. If any underlay link along the path is still at 1500, full-size packets are dropped, while small pings and the TCP handshake pass. I prove it with DF-set pings at increasing sizes, host to host and VTEP loopback to VTEP loopback, then set the fabric MTU to 9216 on NX-OS (9214 on EOS) on every underlay link, including the new or replaced ones.
Go deeper
The math
Inner IP packet 1500 + inner Ethernet header 14 = 1514. Add VXLAN 8, UDP 8 and outer IPv4 20 and the outer IP packet is 1550 bytes. The underlay interface must carry at least that, plus the outer Ethernet header on the wire.
Proving it from a Linux host
ping -M do -s 1472 10.10.10.12 # 1472 + 28 = 1500-byte packet, fails if the underlay is at 1500
ping -M do -s 1422 10.10.10.12 # 1450-byte packet, passesThen test the underlay itself from the leaf: ping the remote VTEP loopback, sourced from the local VTEP
loopback, with the DF bit set and a large size (NX-OS ping 10.0.1.12 source-interface loopback1 df-bit packet-size 8000). Walk the path link by link with show interface | include MTU to find the
odd one.
Why PMTUD does not save you
The underlay switch that drops the packet sends ICMP "fragmentation needed" to the outer source, which is the VTEP, not the host. VTEPs do not translate it back to the inner sender, so the host keeps sending 1500-byte packets into a black hole.
The fix
- Every fabric link (leaf to spine, DCI, border) at a jumbo MTU, the same everywhere.
- Or, where the underlay cannot be raised (some WAN or cloud paths), lower the host MTU to 1450.
They will ask next
Why does SSH log in fine but hang when you run a command with long output?
The login exchange uses small packets. Long output fills full-size segments, which are the ones dropped. The same pattern shows on web pages that load half way.
Where else have you seen 1450-byte host MTUs?
Tenant networks in OpenStack and many Kubernetes CNIs that use VXLAN set 1450 on the guest interface for the same 50-byte reason.
Traps
- Testing with a default 56-byte ping and declaring the path healthy.
- Fixing the leaf uplinks but forgetting a newly added spine or a DCI link.