A server has a 2 x 10G LACP bond but one big transfer never goes above 10 Gbps, and all traffic to the gateway uses one link. Why?
Show the answer
Say this first
LACP balances by flow, not by packet, so one TCP connection always rides one member link and is capped at
10G. On top of that the default xmit_hash_policy is layer2, which hashes only MAC addresses. Every
routed packet has the gateway's MAC as destination, so with layer2 all off-subnet traffic picks the same
link. layer3+4 hashes IPs and ports and spreads many flows across both links. Inbound balance is decided
by the switch's own hash.
Go deeper
See the policy and the per-link load
$ grep -E 'Mode|Hash' /proc/net/bonding/bond0
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
$ ip -s link show ens1f0 | grep -A1 TX
$ ip -s link show ens1f1 | grep -A1 TXOne member's TX counters climbing and the other idle is the symptom.
Change it
ip link set bond0 type bond xmit_hash_policy layer3+4Netplan: transmit-hash-policy: layer3+4 under the bond's parameters. The kernel documentation notes
layer3+4 is not strictly 802.3ad compliant because fragments of one packet may take different links, which
is harmless in practice.
The switch side
Return traffic is balanced by the switch. On a Catalyst, show etherchannel load-balance shows the method;
port-channel load-balance src-dst-ip (or a mixed IP and port method where supported) avoids the same trap
on the way in.
What does give more than 10G to one transfer
Several parallel flows (multi-stream copies, iperf3 -P 4), a faster single link, or mode 0 at the cost of
TCP reordering. LACP never splits one flow.
They will ask next
How would you prove the bond works with iperf3?
Run one stream and see about 9.4 Gbps, then
iperf3 -c 10.10.0.30 -P 8with layer3+4 hashing and watch both members' TX counters move. Several streams can still collide on one link; hashing is statistical.What does it mean if the two slaves show different Aggregator IDs in /proc/net/bonding/bond0?
They did not join the same LACP aggregator, usually because the switch ports are not in one port-channel or are on two switches without MLAG. Only the active aggregator carries traffic, so you have one link of capacity and no redundancy you can count on.
Traps
- Promising 20G for a single flow on a 2 x 10G bond.
- Leaving layer2 hashing on a server whose traffic all goes through one gateway.