TCP, UDP and applications

Side by side

How I decide

• Does late data still have value? A retransmitted voice sample arrives after the jitter buffer has played the gap. For real-time media, a lost packet is better than a late one, so RTP runs on UDP.
• Is the exchange one small question and answer? DNS, NTP and SNMP fit in one datagram each way. A handshake would double the latency for no gain. DHCP has another reason: the client has no IP address yet and must broadcast.
• Do I need one-to-many? Multicast and broadcast only work with UDP.
• Do I want reliability but not TCP's behaviour? QUIC builds streams, retransmission and congestion control on top of UDP in user space, so it can evolve without kernel or middlebox changes.
• Everything else that must be complete and correct goes on TCP.

Fields in order

Options worth knowing by number

TCP has no length field of its own: the segment length is the IP total length minus the IP header minus the TCP data offset.

What each flag means

Common combinations in a capture

• [S] client SYN, [S.] SYN-ACK, [.] pure ACK.
• [P.] data with ACK, the normal data segment.
• [F.] FIN with ACK, a graceful close.
• [R] or [R.] reset. [R.] in reply to a SYN means the port is closed.

Capturing only control segments

tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0'

Fields

Sizes

The largest UDP payload over IPv4 is 65,507 bytes (65,535 minus 20 for IP and 8 for UDP). Anything over the path MTU is fragmented by IP, and losing one fragment loses the whole datagram. That is why DNS over UDP now aims for answers of about 1,232 bytes and QUIC sizes its own packets.

What runs on UDP

DNS 53, DHCP 67 and 68, TFTP 69, NTP 123, SNMP 161 and 162, syslog 514, IKE 500 and NAT-T 4500, RIP 520, BFD 3784, VXLAN 4789, QUIC and HTTP/3 on 443, RTP media on dynamic ports.

What the application must add

Timeouts and retries (DNS, TFTP), sequence numbers (RTP), its own rate control (RFC 8085 sets out guidelines for UDP applications), and keepalives to hold NAT and firewall state open.

The three segments

Client (SYN_SENT)                              Server (LISTEN)
  SYN  seq=x       MSS 1460, SACK ok, TS, wscale 7       ->
                                                (SYN_RCVD)
  <-   SYN,ACK  seq=y ack=x+1   MSS 1460, SACK ok, TS, wscale 7
(ESTABLISHED)
  ACK  seq=x+1 ack=y+1                                   ->
                                                (ESTABLISHED)

The SYN and the SYN-ACK each use up one sequence number, which is why the acknowledgments are ISN plus one.

Why three and not two

Each side has to learn the other's ISN and know that its own ISN was received. Two ISNs need two acknowledgments; the server's SYN and its ACK share one segment, so three segments are enough. The exchange also defeats old duplicate SYNs: if a stale SYN reaches the server, the client answers the unexpected SYN-ACK with a RST instead of an ACK, and no connection opens.

Why the ISN is random

A predictable ISN lets an attacker who cannot see the traffic forge segments into a connection. RFC 6528 describes generating it from a keyed hash of the four-tuple plus a clock.

Options that only exist in the handshake

MSS, window scale and SACK permitted are only valid in SYN and SYN-ACK. If one side does not offer window scaling or SACK, neither side uses it. The third ACK may already carry data.

The sequence

Active closer (A)                         Passive closer (B)
ESTABLISHED                               ESTABLISHED
  FIN  ->
FIN_WAIT_1                                CLOSE_WAIT   (B's app is told: end of stream)
                <-  ACK
FIN_WAIT_2                                 ... B may keep sending data ...
                <-  FIN                   LAST_ACK     (B's app called close())
TIME_WAIT
  ACK  ->                                 CLOSED
(after 2 x MSL) CLOSED

Three segments instead of four

If B's application closes straight away, B sends its FIN in the same segment as the ACK. A capture then shows [F.], [F.], [.]. That is still the same state machine, just with the ACK and FIN combined.

Half-close

Between FIN_WAIT_2 and LAST_ACK, B can still send data and A must still receive and acknowledge it. shutdown(SHUT_WR) uses this on purpose: "I have finished sending, now give me the result".

Simultaneous close

If both sides send FIN at the same time, each goes FIN_WAIT_1, then CLOSING when it receives the other's FIN, then TIME_WAIT when its own FIN is acknowledged. Both sides end in TIME_WAIT.

Port ranges

Defaults differ: Linux uses 32768 to 60999 (net.ipv4.ip_local_port_range), Windows uses 49152 to 65535.

What really limits connections

On the server: memory and file descriptors, not ports. One port can carry millions of connections. On the client: the number of free source ports for each destination IP and port, because the four-tuple must be unique. Behind PAT, every inside host shares the outside address's ports, which is where port exhaustion usually shows up.

Listening versus accepted sockets

ss -tn state established '( sport = :443 )'

Every line has local port 443 and a different peer. The single LISTEN socket is separate and only queues new connections for the application to accept().

Network engineer's list

Protocols that are not ports

These ride directly on IP and are matched by protocol number: ICMP 1, TCP 6, UDP 17, GRE 47, ESP 50, AH 51, EIGRP 88, OSPF 89, VRRP 112.

The picture

bytes:  | acked | sent, not acked | can send now | cannot send yet |
                ^ SND.UNA         ^ SND.NXT      ^ SND.UNA + window

Usable window = min(rwnd, cwnd) minus bytes in flight. When the receiver's application reads data, the receiver advertises a bigger window. When the sender gets ACKs, SND.UNA moves right.

Why it bounds throughput

A sender can put at most one window of data on the wire per round trip, so one flow's throughput is at most window / RTT. That single formula explains most "fast link, slow transfer" tickets.

Who sets what

Round by round

This additive increase and multiplicative decrease (AIMD) makes the familiar sawtooth.

Why the initial window matters

An initial window of 10 segments (RFC 6928) is about 14.6 KB. A small web response fits in the first flight; a large one needs several round trips of slow start, which is why latency, not bandwidth, dominates small transfers.

Slow start after idle

Linux shrinks cwnd back toward the initial window when a connection has been idle for longer than one RTO (net.ipv4.tcp_slow_start_after_idle=1). Long-lived connections that send in bursts, like HTTP/2 or database links, sometimes turn it off.

The layers of a full-size packet

With TCP timestamps on, every segment carries 12 bytes of options, so the data per packet is 1,448 even though the SYN still advertises 1,460. ss -ti shows this as mss:1448.

Common MTUs and matching MSS

MSS is announced, not negotiated

Each side tells the other what it can receive. The two directions can use different sizes. If a SYN has no MSS option, the default is 536 bytes for IPv4 and 1,220 for IPv6.

Side by side

How the client chooses

• HTTP/2: ALPN in the TLS handshake. The client offers h2,http/1.1 and the server picks.
• HTTP/3: the server advertises it, either with an Alt-Svc: h3=":443" response header or a DNS HTTPS record. The browser then tries QUIC and falls back to TCP if UDP 443 is blocked.

RFCs

HTTP semantics RFC 9110, HTTP/1.1 RFC 9112, HTTP/2 RFC 9113 (originally 7540), HTTP/3 RFC 9114, QUIC RFC 9000.

The chain

Root CA            (self-signed, in the OS or browser trust store, not sent by the server)
  signs ->  Intermediate CA   (sent by the server)
              signs ->  Leaf: shop.example.com   (sent by the server, first)

Roots stay offline; intermediates do daily issuing so a compromise can be contained.

Checks, in the order a client roughly applies them

1. Build a path from leaf to a trusted root using what the server sent and the local store.
2. Each certificate's signature verifies with its issuer's public key.
3. Every certificate is within its notBefore and notAfter dates.
4. Issuers have Basic Constraints CA:TRUE, and path length and name constraints are respected.
5. The leaf's Extended Key Usage includes serverAuth.
6. The hostname matches a DNS name in the SAN. Browsers ignore the Common Name.
7. Revocation (CRL or OCSP, often stapled by the server) and, in browsers, certificate transparency proofs.

Wildcards

*.example.com matches shop.example.com, but not example.com and not a.shop.example.com. The wildcard covers exactly one label.

Lifetimes are shrinking

The CA/Browser Forum has voted to cut the maximum lifetime of public TLS certificates in steps, down to 47 days by 2029. Renewal has to be automated, usually with ACME.

Visible versus hidden

How enterprise firewalls filter HTTPS

• Without decryption: by destination IP, SNI and, for TLS 1.2, the certificate.
• With decryption: the firewall acts as a proxy and signs a substitute certificate with an enterprise CA that managed devices trust. Pinned or mutually authenticated applications break and are usually exempted.

HSTS

A Strict-Transport-Security header tells the browser to use HTTPS only for that site for a period, so a later typed http:// link is upgraded before any packet is sent. Preload lists ship that rule in the browser.

Why it is cheap on a server

A server's TIME_WAIT entries all share its listening port and differ by client address and port, so they never block new connections. Linux keeps them in a small structure without buffers.

Where it hurts

A proxy, load balancer or application server connecting to one backend 10.0.0.50:8080 needs a unique source port per connection. Linux's default range is 32768 to 60999, so 28,232 ports. If each connection is closed by the client and waits 60 seconds, the ceiling is about 28,232 / 60, roughly 470 new connections per second to that one backend address and port. Past that, connect() fails with EADDRNOTAVAIL ("Cannot assign requested address").

Fixes, best first

1. Reuse connections: HTTP keep-alive, connection pools, gRPC.
2. Let the server close first, so TIME_WAIT lands on the side that is not short of ports.
3. Add source IPs or backend IPs, since the limit is per four-tuple.
4. Widen net.ipv4.ip_local_port_range.
5. net.ipv4.tcp_tw_reuse=1 lets new outgoing connections reuse a TIME_WAIT four-tuple when timestamps show it is safe. Recent kernels default to 2, which means loopback only.

What not to do

tcp_tw_recycle was removed in Linux 4.12. It dropped SYNs from clients behind NAT whose timestamps did not increase across hosts. Closing with SO_LINGER set to 0 avoids TIME_WAIT by sending RST, and throws away unsent data.

What each pile-up tells you

Naming differs by tool

netstat prints SYN_SENT, ESTABLISHED, CLOSE_WAIT. ss prints SYN-SENT, SYN-RECV, ESTAB, CLOSE-WAIT, TIME-WAIT, and UNCONN for unconnected UDP sockets.

Quick count on Linux

ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

Worked example

Real ISNs are random 32-bit numbers. Small ones keep the arithmetic visible. Client ISN 1000, server ISN 5000.

Rules to say out loud

• Next seq from a sender = this seq + payload length (+1 for SYN or FIN).
• Ack = highest in-order byte received + 1. It never jumps over a hole.
• One ACK can cover many segments (segment 7), and with delayed ACK it usually covers two.

When segment 5 is lost

The client receives segment 6 first. It cannot advance past the hole, so it sends ack 5001 again (a duplicate ACK), with a SACK block 6461-6961 if SACK was agreed. The server sees exactly which bytes to resend.

Common sources of RST

Who really sent it

Compare the RST with the real peer's packets. A RST forged by a middlebox often has a different IP TTL, a different IP ID pattern, or arrives faster than the peer's RTT allows. Capturing on both sides of the firewall settles it.

RST validation

A RST is only accepted if its sequence number matches exactly what the receiver expects next. If it is merely inside the window, the receiver sends a challenge ACK instead (RFC 5961). That makes blind reset attacks much harder.

Example

A Linux host with default buffers sends wscale 7 in its SYN. Later its segments show win 502. The real window is 502 x 128 = 64,256 bytes. Each side picks its own shift from its maximum receive buffer, so the two values can differ.

Rules worth saying

• Only in SYN and SYN-ACK. The factor cannot change during the connection.
• If the client sends it and the SYN-ACK does not, scaling is off in both directions.
• The maximum shift is 14, so the largest window is 65,535 x 16,384, about 1 GiB.

Where it goes wrong

• A capture that starts mid-connection. Wireshark cannot know the factor and shows the raw field; it labels the scaling factor -1 (unknown). Capture from the SYN.
• A middlebox that strips or rewrites the option. The connection works but every flow is capped at 64 KB per RTT. Captures on both sides of the device show the option disappearing.
• An application that sets SO_RCVBUF. On Linux that fixes the buffer and turns off receive autotuning for that socket, which can leave the window far smaller than the path needs.

The formulas

• BDP (bytes) = bandwidth (bit/s) x RTT (s) / 8
• Max throughput of one flow = window (bytes) x 8 / RTT (s)

Worked examples

Loss matters too

Even with big enough windows, loss-based congestion control limits a flow on a lossy long path. The Mathis approximation for Reno-style TCP is throughput = (MSS / RTT) x (1.22 / sqrt(loss)). With MSS 1460, RTT 80 ms and 0.01% loss: (1460 x 8 / 0.08) x (1.22 / 0.01) = 146 kbit/s x 122, about 17.8 Mbit/s. That is why long-haul transfers also need a clean path, or a less loss-sensitive algorithm such as BBR.

What I check on a slow long-distance transfer

1. RTT with ping, and the observed rate. If rate x RTT is close to 64 KB, the window is the limit.
2. ss -ti on the sender: wscale, cwnd, rtt, retrans, and rwnd_limited time.
3. Buffer maximums on both hosts (tcp_rmem, tcp_wmem) against the BDP.

The calculation

• First sample R: SRTT = R, RTTVAR = R / 2.
• Later samples R': RTTVAR = 3/4 RTTVAR + 1/4 abs(SRTT - R'), then SRTT = 7/8 SRTT + 1/8 R'.
• RTO = SRTT + max(G, 4 x RTTVAR), where G is the clock granularity.
• Bounds: the RFC says at least 1 second; Linux uses 200 ms minimum and 120 seconds maximum.

Karn's algorithm

An ACK for a retransmitted segment is ambiguous: it may acknowledge the first copy or the second. TCP takes no RTT sample from it, and keeps the backed-off RTO until a fresh sample arrives. Timestamps remove the ambiguity because the echoed value says which copy is being acknowledged.

When the timer fires

ssthresh = max(flight size / 2, 2 x MSS), cwnd = 1 segment, retransmit the oldest unacknowledged segment, double the RTO. On Linux an established connection gives up after net.ipv4.tcp_retries2 (15) retransmissions, at least about 15 minutes.

Avoiding the timer

Most losses should be repaired by fast retransmit. Linux also sends a tail loss probe about two RTTs after the last segment if nothing has been acknowledged, so a lost final segment triggers recovery without waiting for the full RTO.

Worked example

The sender sends segments 1 to 6, each 1,000 bytes, starting at byte 1. Segment 3 (bytes 2001 to 3000) is lost.

Wireshark labels these TCP Dup ACK and TCP Fast Retransmission.

Fast recovery (RFC 5681, Reno style)

1. ssthresh = max(flight size / 2, 2 x MSS).
2. Retransmit the lost segment.
3. Keep sending new data as duplicate ACKs show segments leaving the network.
4. When new data is acknowledged, set cwnd to ssthresh and continue in congestion avoidance.

NewReno handles several losses in one window. SACK tells the sender exactly which blocks are missing. Linux now detects loss by time with RACK-TLP (RFC 8985) rather than only counting duplicates, which copes better with reordering, and paces the reduction with proportional rate reduction.

What it looks like

10.2.20.30.5201 > 10.1.10.20.51544: Flags [.], ack 1001, win 501, options [nop,nop,TS val 2154883120 ecr 1209364471,nop,nop,sack 1 {2461:5381}], length 0

The receiver has everything up to byte 1000 and also bytes 2461 to 5380. Bytes 1001 to 2460, one 1,460-byte segment, are missing.

Details that come up

• SACK blocks are advisory. The receiver may discard SACKed data, so the sender keeps it until it is cumulatively acknowledged.
• D-SACK (RFC 2883) reports data received twice, which lets the sender detect spurious retransmissions.
• Any device that rewrites sequence numbers, for example a firewall that randomises ISNs, must rewrite SACK edges as well. When one did not, SACK-based recovery stalled.

Two jobs

1. Find dead peers. A peer that crashed or lost its state leaves a half-open connection: our side thinks it is ESTABLISHED forever if we never send. A probe gets a RST or no answer.
2. Keep middlebox state alive. Firewalls and NAT remove idle flows after their idle timeout, often an hour or less. A 2-hour default keepalive is too slow to help; the application's next packet then hits a device with no state and is dropped or reset.

Tuning

System-wide on Linux:

sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes

Per socket, which is better: TCP_KEEPIDLE, TCP_KEEPINTVL and TCP_KEEPCNT. A common rule is an idle time shorter than the shortest idle timeout in the path, for example 300 seconds.

Related

TCP_USER_TIMEOUT caps how long sent data may stay unacknowledged before the connection is dropped, which catches dead peers on busy connections where keepalive never triggers.

Wireshark labels and who sends them

Proving it on the hosts

• Receiver: ss -tn shows a large Recv-Q on that connection: data the kernel holds that the application has not read.
• Sender: ss -tn shows a large Send-Q: data waiting for window.

Why the persist timer exists

If the receiver's window update were lost, both sides would wait forever. The sender's probes force a fresh ACK carrying the current window.

Silly window syndrome

A receiver that advertises every few freed bytes makes the sender send tiny segments. Receivers wait until they can open a meaningful amount (an MSS or half the buffer) before advertising.

Symptoms that shout MTU

• Ping works, the TCP handshake works, small requests work.
• Large transfers stall: web pages load partly, SSH logs in but a long directory listing hangs, file copies freeze at the first big read.
• TLS handshakes stall when the server sends its certificate chain, the first large packet.
• One direction works and the other does not (uploads fine, downloads hang).

How I test from Linux

ping -M do -s 1472 -c 3 10.30.0.10
ping -M do -s 1448 -c 3 10.30.0.10
tracepath -n 10.30.0.10

1472 + 8 ICMP + 20 IP = 1,500. If 1,472 fails with no "Frag needed" message and a smaller size works, something is dropping packets above the real path MTU and the ICMP is not coming back.

Fixes, best first

1. Permit ICMP type 3 code 4 (and ICMPv6 type 2) through every firewall and ACL in the path.
2. Clamp the TCP MSS where the MTU shrinks, such as tunnel interfaces (ip tcp adjust-mss).
3. Set a correct ip mtu on tunnels so the router itself sends a correct ICMP message.
4. Enable packetization layer PMTUD on hosts (net.ipv4.tcp_mtu_probing=1 on Linux), which probes with smaller segments when it detects a black hole instead of relying on ICMP.

Clearing the DF bit on a router makes TCP work by forcing fragmentation, which costs CPU and can break under load. It also cannot help IPv6.

IOS-XE

interface Tunnel0
 ip mtu 1400
 ip tcp adjust-mss 1360
 ipv6 mtu 1400
 ipv6 tcp adjust-mss 1340

ip tcp adjust-mss acts on SYNs entering and leaving the interface, only ever lowers the value, and accepts 500 to 1460. 1360 = 1400 - 20 (IP) - 20 (TCP). For IPv6 subtract 60.

Linux router

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

nftables:

nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu

Junos SRX

set security flow tcp-mss ipsec-vpn mss 1350

What clamping does not cover

• UDP: QUIC, DNS, IPsec from hosts, WireGuard. They need PMTUD or conservative sizes of their own.
• TCP whose SYN takes a different path (asymmetric routing) than the data.
• TCP inside an encrypted tunnel the router only sees from outside.
• Packets that are already large for other reasons, such as non-TCP protocols.

So clamping is a strong safety net, but ICMP should still be allowed.

Features worth naming

• Connection IDs instead of the four-tuple identify a connection, so a phone moving from Wi-Fi to LTE keeps its connection (connection migration).
• Streams are independent: a loss stalls only its own stream.
• Packet numbers never repeat, so retransmissions are not ambiguous for RTT measurement.
• 0-RTT on resumption, with the same replay caveats as TLS 1.3.
• Header protection encrypts packet numbers. On-path devices see little more than the destination connection ID and a few flag bits, including an optional spin bit that lets them estimate RTT.

Sizes and safety

• The client pads its first Initial datagram to at least 1,200 bytes, and QUIC assumes the path carries at least 1,200-byte UDP payloads.
• Before the client's address is validated, a server may send at most three times the bytes it received, which limits amplification attacks.

What the network can and cannot see

Initial packets are encrypted with keys derived from the connection ID that is sent in clear, so any middlebox can decrypt them and read the ClientHello, including the SNI. Everything after the handshake is opaque. A firewall that wants TLS inspection or URL filtering has to block UDP 443 and force a TCP fallback.

The messages

Client                                              Server
ClientHello
  + supported_versions, key_share, signature_algorithms,
    server_name (SNI), ALPN, psk_key_exchange_modes   ->
                                                    ServerHello + key_share
                                                    {EncryptedExtensions}  (ALPN result)
                                                    {Certificate}
                                                    {CertificateVerify}    (signature over the transcript)
                                               <-   {Finished}
{Finished}
[Application data: GET /]                          ->

Braces mean encrypted with handshake keys. The server can send data right after its Finished.

Round trips before the first request byte

What 1.3 removed

RSA key transport (no forward secrecy), static Diffie-Hellman, CBC and RC4 ciphers, SHA-1 in signatures, compression and renegotiation. The cipher suites are AEAD only, for example TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384 and TLS_CHACHA20_POLY1305_SHA256.

Things that look odd in captures

• The version field says TLS 1.2; the real version is in the supported_versions extension. This middlebox compatibility mode also sends a dummy ChangeCipherSpec.
• If the server does not like the client's key share group it sends a HelloRetryRequest, which costs one more round trip.
• Hybrid post-quantum key shares such as X25519MLKEM768 make the ClientHello larger than one packet, which has exposed middleboxes that assumed a ClientHello fits in one segment.

SNI in practice

• Without SNI the server returns its default certificate, and the client reports a name mismatch. Typical culprits: health checks or scripts connecting by IP, and old clients.
• Load balancers and reverse proxies use SNI to choose a certificate and a backend pool, even when they pass the TLS through without terminating it.
• Firewalls use it for URL categories and application identification. Encrypted Client Hello hides the real SNI behind an outer, generic one.

ALPN in practice

• HTTP/2 over TLS is selected only through ALPN h2. A proxy in the middle that offers only http/1.1 downgrades the connection.
• gRPC needs HTTP/2. A TLS-terminating load balancer that does not negotiate h2 toward the client or the backend breaks gRPC calls.
• QUIC uses ALPN too; HTTP/3 is h3.

Testing

openssl s_client -connect 203.0.113.80:443 -servername shop.example.com -alpn h2,http/1.1

The output shows the certificate the server chose for that name and ALPN protocol: h2 when HTTP/2 was selected.

Two queues on a Linux listener

• SYN queue: half-open connections in SYN_RCVD, waiting for the final ACK.
• Accept queue: completed connections waiting for the application to call accept(). Its size is the listen() backlog, capped by net.core.somaxconn (4096 by default since kernel 5.4).

A SYN flood fills the first. A slow application fills the second, which shows as ListenOverflows in nstat and as Recv-Q at or near Send-Q on the LISTEN line of ss -lnt.

How cookies work

The server sets its ISN from a time counter, a small index for the client's MSS, and a cryptographic hash of the four-tuple with a secret. It keeps no state. A real client's ACK carries that ISN plus one; the server checks the hash and the time counter, then creates the connection. Spoofed sources never send the ACK, so they cost the server only an outgoing SYN-ACK.

The trade-off

Only a few MSS values fit in the cookie, and window scale and SACK would be lost. Linux stores those options in the TCP timestamp when timestamps are on, which avoids most of the loss. Linux only switches to cookies when the SYN queue overflows (net.ipv4.tcp_syncookies=1, the default), and the kernel log then shows a "Possible SYN flooding on port ... Sending cookies" message.

Other layers of defence

SYN proxy on a firewall or load balancer, which completes the handshake itself before opening the server side; rate limits; and upstream scrubbing for volumetric floods that fill the link itself.

CUBIC (RFC 9438)

• Window as a function of time t since the last reduction: W(t) = C x (t - K)^3 + Wmax, with C = 0.4.
• Concave as it approaches the old maximum Wmax, flat near it, then convex as it probes beyond it.
• Multiplicative decrease factor 0.7. Growth does not depend on RTT, so long-RTT flows compete better than with Reno.
• Still loss-based: it fills buffers before it backs off, so it adds queueing delay.

BBR (Google, 2016)

• Builds a model: BtlBw, the maximum recent delivery rate, and RTprop, the minimum recent RTT.
• Paces sending at about BtlBw, with cwnd around 2 x BtlBw x RTprop.
• Phases: Startup (gain about 2.89), Drain, ProbeBW (cycles pacing gain 1.25, 0.75, then 1), and ProbeRTT (drops to 4 packets for at least 200 ms if the minimum RTT has not been refreshed for 10 s).
• Random loss that is not caused by congestion barely slows it, which is a big win on long lossy paths.

Cautions

• BBR v1 can take more than its share from CUBIC flows when buffers are shallow and less when buffers are deep, and can cause high retransmission rates. Later versions add a response to loss and ECN.
• Congestion control is sender-side only. Changing a client does not change how a server sends to it.
• BBR relies on pacing. Older kernels needed the fq qdisc; newer kernels pace inside TCP, though fq is still commonly set.

Switching on Linux

sysctl net.ipv4.tcp_available_congestion_control
sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbr

The stall step by step

1. Client writes a 100-byte header. No data is outstanding, so it goes immediately.
2. Client writes a 50-byte body. Data is outstanding and the segment is small, so Nagle holds it.
3. Server gets the header. It cannot answer until it has the body, so there is no reply to piggyback on, and it delays its ACK.
4. The delayed ACK timer fires, the ACK goes, Nagle releases the body, the server answers. Every request pays the timer.

Fixes

• Write the whole message in one call (or use writev), so there is no second small write.
• Set TCP_NODELAY on the sender for request and response or interactive protocols. SSH and many RPC libraries do this; Go sets it by default.
• On Linux the receiver can use TCP_QUICKACK, which is temporary and needs resetting.

Delayed ACK rules

A receiver acknowledges at least every second full-sized segment, and must not delay more than 500 ms (RFC 1122). Linux adapts its delay between about 40 ms and 200 ms.

IP codepoints (2 bits after the 6-bit DSCP)

TCP negotiation

• SYN with ECE and CWR set: "I support ECN".
• SYN-ACK with ECE set: "So do I".
• Data packets then carry ECT. A router marks CE; the receiver sets ECE on its ACKs until the sender replies with CWR.

Where it matters

• Data centers: DCTCP and RoCEv2 congestion control rely on ECN marking at low queue thresholds.
• L4S (RFC 9330 to 9332) uses ECT(1) for very low queueing delay.
• Linux: net.ipv4.tcp_ecn=2 by default (accept, do not initiate); 1 requests ECN on outgoing connections.

Network gotcha

QoS policies that rewrite the whole ToS byte instead of only DSCP wipe ECN bits. Remarking DSCP should leave the two ECN bits alone.

HTTP/1.1

Pipelining lets a client send several requests without waiting, but responses must return in order, so a slow first response blocks the rest. Browsers never enabled it and open about six parallel connections instead.

HTTP/2

Frames from many streams interleave on one TCP connection. The application is no longer blocked, but the kernel will not hand over bytes after a gap. With 1% loss on a busy page, one gap freezes every image, script and API call on the connection for a round trip or more.

HTTP/3 and QUIC

QUIC numbers packets, not bytes of one stream. Each stream has its own offsets, so data for stream 8 can be delivered while stream 4 waits for a retransmission. Order is only kept inside a stream.

The header compression detail

HPACK's dynamic table assumes in-order delivery: every header block may refer to earlier ones. QPACK was designed so that header decoding does not reintroduce cross-stream blocking, using separate encoder and decoder streams and letting the encoder limit how much a stream may depend on unacknowledged table entries.

How it works

1. In a previous connection, the server sent a NewSessionTicket.
2. The new ClientHello carries pre_shared_key and early_data, followed by application data encrypted with the early traffic key.
3. The server accepts by echoing early_data in EncryptedExtensions, or rejects it, in which case the client resends the data after the normal 1-RTT handshake.

Why replay is hard to stop

Within one server, single-use tickets or a cache of recent ClientHellos can catch duplicates. Across a cluster or many data centers, that state would have to be shared, so most deployments accept some replay window and limit what early data may do.

How applications protect themselves

• Only allow safe, idempotent requests in early data, such as GET without side effects.
• A server or CDN that forwards early data to a backend marks it with Early-Data: 1; the backend can answer 425 Too Early to make the client retry after the handshake (RFC 8470).
• nginx has ssl_early_data off by default.

1. Before any packet

• Parse the URL. HSTS or the preload list may force HTTPS for http:// links.
• Check the HTTP cache and service worker; reuse an open connection to the same origin if one exists.

2. Name resolution

• Browser DNS cache, then the OS (hosts file, then the stub resolver), then the configured recursive resolver over UDP 53 or encrypted DNS.
• On a cache miss the resolver asks a root server for .com, a .com TLD server for example.com, and the authoritative server for the A, AAAA and HTTPS records. TTLs govern caching at each step.
• An HTTPS record can announce alpn="h3,h2" and keys for Encrypted Client Hello.

3. Getting the first packet out

• The destination is off-subnet, so the frame goes to the default gateway. ARP (IPv4) or neighbor discovery (IPv6) finds its MAC if not cached.
• Happy Eyeballs (RFC 8305) starts IPv6 first and IPv4 after a short delay, about 250 ms, and keeps whichever connects first.
• The edge firewall does PAT: a private source like 192.168.1.20:51544 becomes a public address and port.
• Routers forward hop by hop on longest prefix match; across providers, BGP chose the path. The service is often anycast, so the nearest CDN site answers.

4. Transport and security

• TCP: SYN with MSS, window scale, SACK and timestamps, then SYN-ACK and ACK: one RTT.
• TLS 1.3: ClientHello with SNI shop.example.com, ALPN h2,http/1.1 and a key share; the server returns its certificate chain encrypted; the client validates the chain and the name: one more RTT.
• Or, if HTTP/3 was advertised: one QUIC handshake on UDP 443 covers both.

5. The request and the server side

• HTTP/2 GET / with :authority, cookies and Accept-Encoding, compressed with HPACK.
• A CDN may answer from cache. Otherwise a layer 7 load balancer terminates TLS, picks a healthy backend, and the application talks to caches and databases.

6. The response and rendering

• Status and headers (Cache-Control, Set-Cookie, Strict-Transport-Security, Alt-Svc), then a compressed body.
• The first flight is limited by the initial congestion window of 10 segments, about 14.6 KB.
• The browser parses HTML, finds CSS, JavaScript and images, fetches them over the same multiplexed connection, builds the DOM and CSSOM, lays out and paints.

7. Afterwards

The connection stays open for reuse. When it finally closes, whichever side sends the first FIN holds TIME_WAIT.