A single file copy over a 1 Gb/s WAN with 80 ms of latency only reaches about 6.5 Mb/s. Why?
Show the answer
Say this first
One TCP flow can send at most one window of data per round trip. To fill 1 Gb/s at 80 ms, the window has to hold the bandwidth delay product: 1 Gb/s x 0.08 s = 80 Mb, which is 10 MB. If window scaling is off or stripped, the window stops at 65,535 bytes, and 65,535 x 8 / 0.08 s is about 6.5 Mb/s. The link is fine; the flow is window limited. The fixes are window scaling with bigger socket buffers, parallel streams, or WAN optimization.
Go deeper
The formula
Maximum throughput of one flow = window / RTT. The window needed to fill a link = bandwidth x RTT.
| Window | RTT | Maximum per flow |
|---|---|---|
| 64 KB | 80 ms | about 6.5 Mb/s |
| 64 KB | 10 ms | about 52 Mb/s |
| 4 MB | 80 ms | about 420 Mb/s |
| 10 MB | 80 ms | about 1 Gb/s |
What keeps the window small
- Window scaling disabled, or stripped by a middlebox. The scale option is only sent in the SYN and SYN-ACK, so a device that rewrites those can cap every flow.
- Host buffer limits: the Linux
net.ipv4.tcp_rmemmaximum, or Windows receive window auto-tuning set to disabled. - Application buffers, for example an old transfer tool with a fixed buffer size.
- Loss: after a loss, congestion control cuts the congestion window, and on a long path it takes many round trips to grow back. CUBIC and BBR cope better than Reno.
Proving it
- Check the handshake: Wireshark shows
WS=in the SYN and SYN-ACK info when scaling is offered. - Statistics, TCP Stream Graphs, Window Scaling: bytes in flight sitting flat at the receive window means the window is the limit.
iperf3 -c 10.30.0.15 -P 8with parallel streams shows whether the capacity exists.
They will ask next
Why do parallel streams help?
Each flow has its own window, so eight flows carry eight windows per round trip. Many backup and transfer tools do this. It also limits the damage from one loss to one flow.
Would a 10 Gb/s circuit help?
No. A window-limited flow gets the same throughput on 1 or 10 Gb/s. Only a bigger window or a shorter round trip helps.
Traps
- Upgrading the circuit for a transfer that is limited by its window.