What is the bandwidth-delay product? Work out the window you need for a 1 Gbit/s path with 80 ms RTT.
Show the answer
Say this first
The bandwidth-delay product is how much data must be in flight to keep a path full: bandwidth times round-trip time. For 1 Gbit/s and 80 ms that is 1,000,000,000 x 0.08 = 80,000,000 bits, which is 10 MB. One TCP flow needs a receive window, a congestion window and a send buffer of at least 10 MB to fill it. With an unscaled 64 KB window the same flow tops out at 65,535 x 8 / 0.08, about 6.5 Mbit/s.
Go deeper
The formulas
- BDP (bytes) = bandwidth (bit/s) x RTT (s) / 8
- Max throughput of one flow = window (bytes) x 8 / RTT (s)
Worked examples
| Path | BDP | 64 KB window gives |
|---|---|---|
| 1 Gbit/s, 1 ms (inside a data center) | 125 KB | about 524 Mbit/s |
| 100 Mbit/s, 20 ms | 250 KB | about 26 Mbit/s |
| 1 Gbit/s, 80 ms | 10 MB | about 6.5 Mbit/s |
| 10 Gbit/s, 100 ms | 125 MB | about 5.2 Mbit/s |
Loss matters too
Even with big enough windows, loss-based congestion control limits a flow on a lossy long path. The Mathis approximation for Reno-style TCP is throughput = (MSS / RTT) x (1.22 / sqrt(loss)). With MSS 1460, RTT 80 ms and 0.01% loss: (1460 x 8 / 0.08) x (1.22 / 0.01) = 146 kbit/s x 122, about 17.8 Mbit/s. That is why long-haul transfers also need a clean path, or a less loss-sensitive algorithm such as BBR.
What I check on a slow long-distance transfer
- RTT with ping, and the observed rate. If rate x RTT is close to 64 KB, the window is the limit.
ss -tion the sender:wscale,cwnd,rtt,retrans, andrwnd_limitedtime.- Buffer maximums on both hosts (
tcp_rmem,tcp_wmem) against the BDP.
They will ask next
Why do parallel streams make a slow transfer faster?
Each stream has its own window. If one flow is window-limited at 6.5 Mbit/s, eight flows get about 52 Mbit/s. That is a workaround that proves the diagnosis. The real fix is a window that covers the BDP.
Does a bigger buffer on the receiver always help?
Only up to the BDP. Beyond that the sender just fills router queues, which adds latency (bufferbloat). The sender's send buffer also has to hold a full window of unacknowledged data.
Traps
- Forgetting to convert bits to bytes. 1 Gbit/s x 80 ms is 80 Mbit, which is 10 MB, not 80 MB.
- Using one-way delay instead of round-trip time.