Set the IP MTU on a GRE tunnel and clamp TCP MSS so large transfers work

Tasks

1. Build a GRE tunnel between R1 and R2 and route each site's LAN over it.
2. Set the tunnel IP MTU to 1,400 bytes on both routers.
3. Clamp the TCP MSS on the tunnel to match the IP MTU.
4. Prove the MTU with a DF ping from the router and PMTUD from PC1.
5. Prove from a capture on SRV1 that TCP SYNs crossing the tunnel carry the clamped MSS.

Solution

1. 1Interfaces and tunnel on R1R1

   interface GigabitEthernet0/0/0
    ip address 198.51.100.2 255.255.255.252
    no shutdown
   interface GigabitEthernet0/0/1
    ip address 10.1.0.1 255.255.255.0
    no shutdown
   interface Tunnel0
    ip address 172.16.0.1 255.255.255.252
    ip mtu 1400
    ip tcp adjust-mss 1360
    tunnel source GigabitEthernet0/0/0
    tunnel destination 203.0.113.2

   GRE over IPv4 is the default tunnel mode. ip mtu 1400 is the largest IP packet R1 puts into the tunnel before encapsulation; 1360 = 1400 - 20 (IP) - 20 (TCP).

2. 2Routing on R1R1

   ip route 0.0.0.0 0.0.0.0 198.51.100.1
   ip route 10.2.0.0 255.255.255.0 172.16.0.2

   The tunnel destination is reached by the default route through Gi0/0/0, never through the tunnel itself. Otherwise the tunnel flaps with %TUN-5-RECURDOWN (recursive routing).

3. 3Interfaces and tunnel on R2R2

   interface GigabitEthernet0/0/0
    ip address 203.0.113.2 255.255.255.252
    no shutdown
   interface GigabitEthernet0/0/1
    ip address 10.2.0.1 255.255.255.0
    no shutdown
   interface Tunnel0
    ip address 172.16.0.2 255.255.255.252
    ip mtu 1400
    ip tcp adjust-mss 1360
    tunnel source GigabitEthernet0/0/0
    tunnel destination 198.51.100.2

4. 4Routing on R2R2

   ip route 0.0.0.0 0.0.0.0 203.0.113.1
   ip route 10.1.0.0 255.255.255.0 172.16.0.1

Verify

R1# show ip interface Tunnel0 | include line protocol|MTU
Tunnel0 is up, line protocol is up
  MTU is 1400 bytes

R1# ping 10.2.0.10 source 10.1.0.1 size 1400 df-bit
Type escape sequence to abort.
Sending 5, 1400-byte ICMP Echos to 10.2.0.10, timeout is 2 seconds:
Packet sent with a source address of 10.1.0.1
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 21/22/24 ms

PC1# ping -M do -s 1373 -c 2 10.2.0.10
PING 10.2.0.10 (10.2.0.10) 1373(1401) bytes of data.
From 10.1.0.1 icmp_seq=1 Frag needed and DF set (mtu = 1400)
ping: local error: message too long, mtu=1400

--- 10.2.0.10 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms

SRV1# sudo tcpdump -ni eth0 'tcp[tcpflags] & tcp-syn != 0 and host 10.1.0.10'
10:21:07.501933 IP 10.1.0.10.40122 > 10.2.0.10.22: Flags [S], seq 1730514481, win 64240, options [mss 1360,sackOK,TS val 935713092 ecr 0,nop,wscale 7], length 0
10:21:07.501981 IP 10.2.0.10.22 > 10.1.0.10.40122: Flags [S.], seq 4092375513, ack 1730514482, win 65160, options [mss 1460,sackOK,TS val 2771039415 ecr 935713092,nop,wscale 7], length 0

Mistakes people make

• Using mtu instead of ip mtu on the tunnel. ip mtu sets the IP packet size, which is what PMTUD and fragmentation use.
• Setting ip tcp adjust-mss to the MTU value (1400). The MSS is payload only; subtract 40 bytes for IPv4 (60 for IPv6).
• Letting the route to the tunnel destination point into the tunnel, which causes recursive routing and a flapping tunnel.
• An ACL on the internet interfaces that does not permit GRE (IP protocol 47) between the two public addresses.
• Expecting MSS clamping to help UDP traffic such as QUIC or DNS. Those still rely on PMTUD, so ICMP type 3 code 4 must be allowed.

On other platforms

ip tunnel add gre1 mode gre local 198.51.100.2 remote 203.0.113.2 ttl 255
ip addr add 172.16.0.1/30 dev gre1
ip link set gre1 mtu 1400 up
ip route add 10.2.0.0/24 via 172.16.0.2
iptables -t mangle -A FORWARD -o gre1 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu