A spine fails, or you need to upgrade one. What happens to traffic, and how do you take a spine out of service cleanly?
Show the answer
Say this first
On a hard failure the leaf sees the link go down, tears down the directly connected eBGP session at once and removes that spine from the ECMP group. Flows rehash onto the remaining spines in well under a second. If the link stays up but the spine stops forwarding, BFD detects it. For an upgrade I drain first: make the spine's paths less attractive (AS-path prepend, graceful shutdown community, or NX-OS maintenance mode), watch traffic move off, then reload.
Go deeper
Failure detection
- Link down. Fastest. Directly connected eBGP sessions drop immediately when the interface goes down.
- BFD. Catches failures where light stays on: a hung line card, a one-way fiber, a DWDM or media converter in the middle. Typical values are a few hundred milliseconds total detection.
- BGP hold timer. 180 s by default. Never rely on it in a fabric.
Clean drain options
- AS-path prepend outbound on the spine toward every leaf. Leaves still have the path as a backup.
- GRACEFUL_SHUTDOWN community
65535:0(RFC 8326). Receivers that match it set local preference to 0. - Vendor maintenance mode. NX-OS Graceful Insertion and Removal (
system mode maintenance) isolates the switch from routing protocols. EOS has its own maintenance mode for BGP peers and interfaces.
Upgrade routine
- Check the remaining spines can carry the load (oversubscription with N minus one).
- Drain, then confirm interface counters on the spine drop to near zero.
- Upgrade and reload, verify sessions and routes, then undrain.
They will ask next
Why drain at all if ECMP fails over in under a second?
A reload is not one clean failure. Ports come up before routing is ready, and routes can be advertised before the forwarding tables are programmed. That causes black holes. Draining moves traffic before anything breaks, and the undrain step lets the switch settle first.
Traps
- Upgrading two spines at once in a four-spine fabric and halving capacity at peak time.