Size Linux TCP buffers for a 1 Gbit/s path with 60 ms RTT

LinuxWorking

Make a single TCP flow fill a 1 Gbit/s path with a 60 ms round trip: work out the bandwidth-delay product, raise the TCP buffer limits on both hosts to cover it, optionally switch the sender to BBR, and prove the result with iperf3 and ss.

Item	Value
Path bandwidth	1 Gbit/s
Round-trip time	about 60 ms
Sender	SRV1, 10.1.10.20
Receiver	BKP1, 10.2.20.30, runs `iperf3 -s`

Both hosts run a recent Linux kernel with default sysctl values. Window scaling works end to end.

Tasks

Measure the RTT and calculate the bandwidth-delay product.
Read the current TCP buffer limits on both hosts and compare them with the BDP.
Raise the limits persistently on both hosts.
Switch the sender to BBR with the fq queueing discipline.
Prove that one stream now runs close to line rate.

Solution

1Measure the RTT and work out the BDPSRV1
SRV1 on Linux
```
ping -c 5 10.2.20.30
# BDP = 1,000,000,000 bit/s x 0.060 s / 8 = 7,500,000 bytes (7.5 MB)
```
One flow needs a receive window, a congestion window and a send buffer of at least 7.5 MB. Linux advertises only part of its receive buffer as window, so size the maximum well above the BDP.
2Read the current limitsBKP1
BKP1 on Linux
```
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.core.rmem_max net.core.wmem_max
```
tcp_rmem and tcp_wmem hold minimum, default and maximum in bytes. Autotuning works between them. rmem_max and wmem_max only cap applications that set SO_RCVBUF or SO_SNDBUF themselves.
3Raise the limits on both hostsSRV1
SRV1 on Linux
```
# /etc/sysctl.d/90-wan-buffers.conf (same file on BKP1)
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 131072 33554432
net.ipv4.tcp_wmem = 4096 16384 33554432
```
Apply with sudo sysctl --system. 32 MB is about four times the BDP. Only the maximum changes; the default stays small so thousands of idle sockets do not each reserve megabytes. Settings apply to new connections only.
4Switch the sender to BBR with fqSRV1
SRV1 on Linux
```
sudo modprobe tcp_bbr
sysctl net.ipv4.tcp_available_congestion_control
# add to /etc/sysctl.d/90-wan-buffers.conf on SRV1
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
```
Congestion control only matters on the sending side, so BKP1 can stay on CUBIC. On a clean path CUBIC also fills the pipe once buffers are right; BBR helps most when the path has some random loss.
5Run the testSRV1
SRV1 on Linux
```
iperf3 -c 10.2.20.30 -t 10
ss -ti dst 10.2.20.30
```
Run ss -ti in a second terminal while iperf3 is running.

Verify

SRV1 output

SRV1# ping -c 5 10.2.20.30
PING 10.2.20.30 (10.2.20.30) 56(84) bytes of data.
64 bytes from 10.2.20.30: icmp_seq=1 ttl=62 time=60.4 ms
64 bytes from 10.2.20.30: icmp_seq=2 ttl=62 time=60.1 ms
64 bytes from 10.2.20.30: icmp_seq=3 ttl=62 time=60.2 ms
64 bytes from 10.2.20.30: icmp_seq=4 ttl=62 time=60.1 ms
64 bytes from 10.2.20.30: icmp_seq=5 ttl=62 time=60.3 ms

--- 10.2.20.30 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 60.104/60.233/60.418/0.112 ms

RTT is about 60 ms and TTL 62 shows two routed hops (R1 and R2). The BDP is 7.5 MB.

BKP1 output

BKP1# sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.core.rmem_max net.core.wmem_max
net.ipv4.tcp_rmem = 4096	131072	6291456
net.ipv4.tcp_wmem = 4096	16384	4194304
net.core.rmem_max = 212992
net.core.wmem_max = 212992

Before the change: a 6 MB receive maximum and a 4 MB send maximum, both below the 7.5 MB BDP. A single flow cannot fill the path.

SRV1 output

SRV1# iperf3 -c 10.2.20.30 -t 10
Connecting to host 10.2.20.30, port 5201
[  5] local 10.1.10.20 port 41870 connected to 10.2.20.30 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  58.2 MBytes   488 Mbits/sec    0   13.9 MBytes
[  5]   1.00-2.00   sec   111 MBytes   931 Mbits/sec    0   14.0 MBytes
[  5]   2.00-3.00   sec   111 MBytes   931 Mbits/sec    0   14.0 MBytes
[  5]   3.00-4.00   sec   111 MBytes   932 Mbits/sec    0   14.0 MBytes
[  5]   4.00-5.00   sec   111 MBytes   931 Mbits/sec    0   14.0 MBytes
[  5]   5.00-6.00   sec   111 MBytes   932 Mbits/sec    0   14.0 MBytes
[  5]   6.00-7.00   sec   111 MBytes   931 Mbits/sec    0   14.0 MBytes
[  5]   7.00-8.00   sec   111 MBytes   932 Mbits/sec    0   14.0 MBytes
[  5]   8.00-9.00   sec   111 MBytes   931 Mbits/sec    0   14.0 MBytes
[  5]   9.00-10.00  sec   111 MBytes   932 Mbits/sec    0   14.0 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.03 GBytes   886 Mbits/sec    0             sender
[  5]   0.00-10.06  sec  1.03 GBytes   881 Mbits/sec                  receiver

iperf Done.

After the first second of startup, one stream holds about 930 Mbit/s, close to the most TCP can carry over 1 Gbit/s Ethernet once headers are counted. BBR keeps cwnd near twice the BDP (about 14 MB) and paces at the bottleneck rate, with no retransmissions.

SRV1 output

SRV1# ss -ti dst 10.2.20.30
State Recv-Q Send-Q   Local Address:Port   Peer Address:Port Process
ESTAB 0      14530176    10.1.10.20:41870    10.2.20.30:5201
	 ts sack bbr wscale:10,10 rto:264 rtt:60.48/0.21 ato:40 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10138 bytes_sent:595591168 bytes_acked:581061168 segs_out:411320 segs_in:98114 data_segs_out:411318 bbr:(bw:931Mbps,mrtt:60.02,pacing_gain:1,cwnd_gain:2) send 1.94Gbps lastrcv:5130 pacing_rate 922Mbps delivery_rate 931Mbps delivered:401287 busy:5130ms unacked:10031 rcv_space:14480 rcv_ssthresh:64088 minrtt:60.02

bbr is in use, and its model in bbr:(...) shows a bottleneck bandwidth of 931 Mbit/s and a 60 ms minimum RTT. wscale:10,10 shows both sides agreed a shift of 10, which follows from the 32 MB buffer maximum. cwnd:10138 segments of 1,448 bytes is about 14 MB. There is no rwnd_limited field at all, so the receiver is no longer the limit.

Mistakes people make

Raising only net.core.rmem_max. TCP autotuning uses the third value of net.ipv4.tcp_rmem; the core value only caps applications that set their own buffer.
Tuning only the receiver. The sender's tcp_wmem maximum must also hold a full BDP of unacknowledged data.
Raising the middle (default) value to megabytes. Every socket then starts with a large buffer and memory use grows on busy servers.
Testing with an application that sets SO_RCVBUF itself. That disables autotuning for its sockets, so sysctl changes to tcp_rmem do not help it.
Expecting BBR on the receiver to speed up a download. Congestion control is a sender-side choice.

On other platforms

Check the effect per connection without rebooting anything, and see the cached path metrics.

Linux

ss -tin state established '( dport = :5201 )'
ip tcp_metrics show 10.2.20.30

Say it in the interview

"A 1 Gbit/s path with 60 ms RTT has a BDP of 7.5 MB, and the default Linux maximums of 6 MB receive and 4 MB send are below that. I raised the tcp_rmem and tcp_wmem maximums to 32 MB on both hosts, left the defaults small, and switched the sender to BBR with fq. iperf3 then showed one stream at about 930 Mbit/s, and ss showed a window scale of 10 and a congestion window around twice the BDP."

Book 1.5 Performance Book 6.3 TCP Congestion Control Book 6.4 Advanced Congestion Control