How does ECMP work in a spine-leaf fabric, and what can go wrong with it?
Show the answer
Say this first
Each leaf learns every remote prefix from every spine at equal cost and installs all of them as next hops. For each packet it hashes header fields, normally the 5-tuple, and the hash picks one uplink, so a flow always takes the same path and stays in order. Problems: it balances flows, not bytes, so a few elephant flows can overload one link; identical hashing at every tier can polarize; and BGP installs only one path unless you raise maximum-paths.
Go deeper
Making it work
- BGP installs one best path by default on IOS, NX-OS and EOS. Configure
maximum-paths(eBGP) to at least the number of spines. On EOS it ismaximum-paths 4 ecmp 4. - With OSPF or IS-IS, equal-cost paths are installed automatically up to the platform's limit.
- VXLAN keeps ECMP useful: the VTEP sets the outer UDP source port from a hash of the inner headers, so different inner flows between the same two VTEPs take different spines.
What goes wrong
- Elephant flows. Backups, storage rebuilds and replication are few and huge. Two of them can land on the same 100G uplink while others idle. Fixes: more or faster uplinks, more parallel streams, or dynamic load balancing (flowlet switching) where the ASIC supports it.
- Polarization. If every switch hashes the same fields with the same function, flows that chose link 1 at one stage all make the same choice at the next stage. Vendors add a per-switch seed.
- Rehash on change. Adding or removing a path can move many flows. Resilient hashing moves only the flows that used the failed path.
- Unequal paths. One leaf with two links to a spine and one link to another breaks the "equal" in ECMP. Keep cabling symmetric.
They will ask next
Why not balance per packet?
Packets of one flow would take paths with different queue depths and arrive out of order. TCP reads reordering as loss, sends duplicate ACKs, retransmits and slows down.
How do you find which uplink a flow uses?
Use the platform's hash calculator (on NX-OS
show routing hash), or send a test flow and watch interface counters. In VXLAN, remember the hash is on the outer header.
Traps
- Saying ECMP gives each flow the sum of all links. One flow is limited to one link.
- Forgetting
maximum-paths, then wondering why only one spine carries traffic.