Build an eBGP underlay for a two-spine, two-leaf fabric

Tasks

1. Configure routed /31 fabric links with MTU 9216 on every spine and leaf interface.
2. Create loopback0 on all switches and loopback1 on the leaves.
3. Run eBGP from each leaf to both spines, with BFD on every session.
4. Advertise only the loopbacks into BGP.
5. Enable ECMP so each leaf installs both spines as next hops.

Solution

1. 1Fabric interfaces and loopbacks on LEAF1LEAF1

   feature bgp
   feature bfd
   !
   interface Ethernet1/49
     description to SPINE1 Eth1/1
     no switchport
     mtu 9216
     ip address 10.1.1.1/31
     no ip redirects
     no shutdown
   interface Ethernet1/50
     description to SPINE2 Eth1/1
     no switchport
     mtu 9216
     ip address 10.1.2.1/31
     no ip redirects
     no shutdown
   interface loopback0
     description router ID and BGP
     ip address 10.0.0.11/32
   interface loopback1
     description VTEP
     ip address 10.0.1.11/32

   NX-OS requires no ip redirects on interfaces that run BFD. LEAF2 is the same with its own addresses.

2. 2eBGP to both spines on LEAF1LEAF1

   router bgp 65001
     router-id 10.0.0.11
     address-family ipv4 unicast
       network 10.0.0.11/32
       network 10.0.1.11/32
       maximum-paths 4
     template peer SPINES
       remote-as 65100
       bfd
       address-family ipv4 unicast
     neighbor 10.1.1.0
       inherit peer SPINES
       description SPINE1
     neighbor 10.1.2.0
       inherit peer SPINES
       description SPINE2

   The template keeps leaf configs short and identical. maximum-paths 4 allows up to four eBGP paths; without it only one spine is used. redistribute direct route-map ... matching the loopbacks works too, but explicit network statements make it obvious what the leaf advertises.

3. 3Fabric interfaces and BGP on SPINE1SPINE1

   feature bgp
   feature bfd
   !
   interface Ethernet1/1
     description to LEAF1 Eth1/49
     no switchport
     mtu 9216
     ip address 10.1.1.0/31
     no ip redirects
     no shutdown
   interface Ethernet1/2
     description to LEAF2 Eth1/49
     no switchport
     mtu 9216
     ip address 10.1.1.2/31
     no ip redirects
     no shutdown
   interface loopback0
     ip address 10.0.0.1/32
   !
   router bgp 65100
     router-id 10.0.0.1
     address-family ipv4 unicast
       network 10.0.0.1/32
       maximum-paths 4
     neighbor 10.1.1.1
       remote-as 65001
       description LEAF1
       bfd
       address-family ipv4 unicast
     neighbor 10.1.1.3
       remote-as 65002
       description LEAF2
       bfd
       address-family ipv4 unicast

   Each leaf has its own ASN, so the spine lists neighbors one by one. SPINE2 is identical with 10.1.2.x addresses and router ID 10.0.0.2. In large fabrics these spine stanzas are generated from a source of truth rather than typed.

Verify

LEAF1# show bgp ipv4 unicast summary
BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.0.0.11, local AS number 65001
BGP table version is 12, IPv4 Unicast config peers 2, capable peers 2

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.1.0        4 65100      37      35       12    0    0 00:24:10 3
10.1.2.0        4 65100      36      35       12    0    0 00:24:07 3

LEAF1# show ip route 10.0.1.12
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.1.12/32, ubest/mbest: 2/0
    *via 10.1.1.0, [20/0], 00:24:01, bgp-65001, external, tag 65100
    *via 10.1.2.0, [20/0], 00:24:01, bgp-65001, external, tag 65100

LEAF1# ping 10.0.1.12 source-interface loopback1 count 3
PING 10.0.1.12 (10.0.1.12) from 10.0.1.11: 56 data bytes
64 bytes from 10.0.1.12: icmp_seq=0 ttl=254 time=1.012 ms
64 bytes from 10.0.1.12: icmp_seq=1 ttl=254 time=0.734 ms
64 bytes from 10.0.1.12: icmp_seq=2 ttl=254 time=0.701 ms

--- 10.0.1.12 ping statistics ---
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min/avg/max = 0.701/0.815/1.012 ms

Mistakes people make

• Leaving maximum-paths at the default of 1. Only one spine carries traffic and the other is idle backup.
• Giving every spine its own ASN, which reintroduces path hunting and lets routes pass through two spines.
• Putting all leaves in the same ASN without changing the design. Leaves then drop each other's routes as AS loops.
• Forgetting MTU on a new link. The underlay pings fine with small packets and VXLAN breaks later.
• Advertising every /31 into BGP when only the loopbacks are needed. It bloats the table for no benefit.

On other platforms

ip routing
!
interface Ethernet49/1
   description to SPINE1 Ethernet1
   no switchport
   mtu 9214
   ip address 10.1.1.1/31
interface Ethernet50/1
   description to SPINE2 Ethernet1
   no switchport
   mtu 9214
   ip address 10.1.2.1/31
interface Loopback0
   ip address 10.0.0.11/32
interface Loopback1
   ip address 10.0.1.11/32
!
router bgp 65001
   router-id 10.0.0.11
   maximum-paths 4 ecmp 4
   neighbor SPINES peer group
   neighbor SPINES remote-as 65100
   neighbor SPINES bfd
   neighbor 10.1.1.0 peer group SPINES
   neighbor 10.1.2.0 peer group SPINES
   network 10.0.0.11/32
   network 10.0.1.11/32