What are the wiring rules in a spine-leaf fabric, and how do you scale it?
Show the answer
Say this first
Every leaf connects to every spine. Spines never connect to each other, leaves never connect to each other except an MLAG or vPC peer link, and all endpoints attach to leaves. The number of spines is capped by uplinks per leaf, and the number of leaves is capped by ports per spine. Add leaves for more server ports, add spines for more bandwidth, and go to a five-stage design with super-spines when spine ports run out.
Go deeper
Worked example
Leaves with 48 x 25G down and 6 x 100G up, spines with 64 x 100G:
- Up to 6 spines (one uplink from each leaf to each spine).
- Up to 64 leaves (one spine port per leaf), minus any ports used for border leaves.
- 64 leaves x 48 ports = 3,072 server ports at 25G, or 1,536 dual-homed servers.
How it grows
- More ports: add a leaf (or leaf pair) and cable it to every spine. No change to existing racks.
- More bandwidth: add a spine and one uplink from every leaf. Each leaf gains a new ECMP path.
- Past one spine's port count: build pods (each a spine-leaf), then connect pod spines to a super-spine layer. That is a five-stage Clos. Traffic between pods takes four hops.
Roles on the leaves
Not every leaf has servers. A border leaf connects WAN routers or the internet edge, and a service leaf connects firewalls and load balancers. They are still leaves and follow the same rules.
They will ask next
Why no links between spines?
They add nothing. Traffic between two leaves always goes up one spine and down. Spine-to-spine links would create unequal paths and give the routing protocol alternate paths to hunt through during convergence.
What happens when a spine fails?
Each leaf loses one of its ECMP next hops. With four spines, fabric capacity drops 25% and flows on the failed spine rehash onto the others. That is why you size uplinks for N minus one spines.
Traps
- Using a different number of uplinks from different leaves to the same spine. ECMP assumes equal paths.
- Forgetting border leaves when counting spine ports.