Compare loss-based congestion control like Reno and CUBIC with BBR. When would you switch a server to BBR?
Show the answer
Say this first
Reno and CUBIC treat packet loss as the congestion signal: they grow the window until a queue overflows, then cut it. CUBIC, the Linux default, grows as a cubic function of time since the last loss, so it recovers quickly on long fat paths, and cuts to 0.7 of the window on loss. BBR does not react to loss by design: it measures the bottleneck bandwidth and the minimum RTT, paces at that rate and keeps about one to two BDPs in flight. I would try BBR on servers sending over long, slightly lossy paths, such as video or downloads to the internet, after testing fairness with existing traffic.
Go deeper
CUBIC (RFC 9438)
- Window as a function of time t since the last reduction: W(t) = C x (t - K)^3 + Wmax, with C = 0.4.
- Concave as it approaches the old maximum Wmax, flat near it, then convex as it probes beyond it.
- Multiplicative decrease factor 0.7. Growth does not depend on RTT, so long-RTT flows compete better than with Reno.
- Still loss-based: it fills buffers before it backs off, so it adds queueing delay.
BBR (Google, 2016)
- Builds a model: BtlBw, the maximum recent delivery rate, and RTprop, the minimum recent RTT.
- Paces sending at about BtlBw, with cwnd around 2 x BtlBw x RTprop.
- Phases: Startup (gain about 2.89), Drain, ProbeBW (cycles pacing gain 1.25, 0.75, then 1), and ProbeRTT (drops to 4 packets for at least 200 ms if the minimum RTT has not been refreshed for 10 s).
- Random loss that is not caused by congestion barely slows it, which is a big win on long lossy paths.
Cautions
- BBR v1 can take more than its share from CUBIC flows when buffers are shallow and less when buffers are deep, and can cause high retransmission rates. Later versions add a response to loss and ECN.
- Congestion control is sender-side only. Changing a client does not change how a server sends to it.
- BBR relies on pacing. Older kernels needed the
fqqdisc; newer kernels pace inside TCP, thoughfqis still commonly set.
Switching on Linux
sysctl net.ipv4.tcp_available_congestion_control
sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbrThey will ask next
What is bufferbloat and how do these algorithms relate to it?
Bufferbloat is excess latency from oversized queues. Loss-based algorithms fill any queue before they see a drop, so latency rises under load. BBR aims to keep queues near empty. AQM such as fq_codel on the bottleneck attacks the same problem from the network side.
What does DCTCP do differently in a data center?
Switches mark packets with ECN at a low queue threshold. DCTCP cuts its window in proportion to the fraction of marked packets instead of halving, keeping queues short with high throughput. It needs ECN on every switch and host, so it suits a controlled data center, not the internet.
Traps
- Saying BBR fixes a receiver-side window limit. If rwnd or the send buffer is the limit, no congestion control helps.