What is TCP incast, and how do data center switches and hosts deal with it?
Show the answer
Say this first
Incast is many senders answering one receiver at the same instant, as in scatter-gather requests or distributed storage reads. The responses converge on one leaf port, overflow its buffer, and many flows lose packets together. Each waits for a retransmission timeout, which on Linux is at least 200 ms against microsecond round trips, so throughput collapses. Fixes are deeper or better-managed buffers, ECN with DCTCP, a lower minimum RTO, and spreading requests in the application.
Go deeper
Why it hurts so much
- Data center RTTs are tens to hundreds of microseconds. A 200 ms RTO stall is a thousand times longer.
- Losses are synchronized, so many flows stall together and the link sits idle.
- Five-minute or even one-minute interface counters average it away. These are microbursts.
Network side
- Buffers. Shallow-buffer switches share a few tens of MB across all ports. Deep-buffer platforms at congestion points (storage leaves, border, DCI) absorb bursts. Tune the shared-buffer thresholds.
- ECN. Switches mark packets instead of dropping when a queue passes a threshold.
- Telemetry. Watch per-queue drops and buffer occupancy, not interface averages.
Host side
- DCTCP reacts to the fraction of ECN-marked packets with a proportional slowdown, keeping queues short.
- Lower RTO minimum on Linux per route (
ip route change ... rto_min 5ms) in controlled environments. - Application pacing. Add jitter or limit how many servers answer at once.
RDMA is different
RoCEv2 (UDP 4791) does not tolerate loss well, so it runs on lossless classes with PFC (802.1Qbb) plus ECN.
They will ask next
What is the risk of PFC?
Pause frames spread backwards. A slow receiver can pause upstream ports, causing head-of-line blocking, pause storms or deadlock in a looped path. Use PFC only for the classes that need it and enable the PFC watchdog.
Traps
- Diagnosing "no congestion" from interface utilization graphs. Microbursts do not show up there.