Octet

Configure a VXLAN BGP EVPN leaf with an anycast gateway

NX-OSExpert

Tasks

  1. Enable the EVPN and VXLAN features and set the fabric anycast gateway MAC.
  2. Map VLAN 10 to VNI 10010 and VLAN 3001 to L3VNI 50001.
  3. Create VRF TENANT-A with its L3VNI and route targets.
  4. Create the anycast gateway SVI for VLAN 10 and the L3VNI SVI.
  5. Build the NVE interface with BGP host reachability, ingress replication and ARP suppression.
  6. Define the EVPN instance for VNI 10010 with RD and RTs.
  7. Bring up eBGP L2VPN EVPN sessions to both spines and advertise the tenant subnets.
  8. Make sure the spines keep route targets and next hops unchanged.

Solution

  1. 1Features and anycast gateway MACLEAF1
    LEAF1 on NX-OS
    nv overlay evpn
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
feature fabric forwarding
!
fabric forwarding anycast-gateway-mac 2020.0000.00aa
  2. 2VLAN to VNI mapping and the tenant VRFLEAF1
    LEAF1 on NX-OS
    vlan 10
  name WEB
  vn-segment 10010
vlan 3001
  name L3VNI-TENANT-A
  vn-segment 50001
!
vrf context TENANT-A
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both 65000:50001
    route-target both 65000:50001 evpn

    The evpn keyword sets the route targets used for EVPN routes in this VRF (type 2 host routes and type 5 prefixes).

  3. 3Gateway SVI, L3VNI SVI and the server portLEAF1
    LEAF1 on NX-OS
    interface Vlan10
  no shutdown
  vrf member TENANT-A
  ip address 10.10.10.1/24
  fabric forwarding mode anycast-gateway
interface Vlan3001
  no shutdown
  mtu 9216
  vrf member TENANT-A
  ip forward
interface Ethernet1/1
  description SRV1
  switchport
  switchport mode access
  switchport access vlan 10
  spanning-tree port type edge
  no shutdown

    The L3VNI SVI has no IP address. ip forward lets it route, and it gives the VRF its forwarding context.

  4. 4NVE interfaceLEAF1
    LEAF1 on NX-OS
    interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    suppress-arp
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
  5. 5EVPN instance for the L2VNILEAF1
    LEAF1 on NX-OS
    evpn
  vni 10010 l2
    rd auto
    route-target import 65000:10010
    route-target export 65000:10010

    Manual route targets are used because every leaf has its own ASN. Auto RTs would be 65001:10010 here and 65002:10010 on LEAF2, and nothing would import.

  6. 6eBGP EVPN sessions and tenant prefixesLEAF1
    LEAF1 on NX-OS
    route-map RM-TENANT-SUBNETS permit 10
!
router bgp 65001
  template peer SPINE-EVPN
    remote-as 65100
    update-source loopback0
    ebgp-multihop 2
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.0.0.1
    inherit peer SPINE-EVPN
    description SPINE1 overlay
  neighbor 10.0.0.2
    inherit peer SPINE-EVPN
    description SPINE2 overlay
  vrf TENANT-A
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map RM-TENANT-SUBNETS

    Extended communities carry the route targets, the encapsulation and the router MAC. Without send-community extended, routes arrive but are never imported. The redistribute line exports 10.10.10.0/24 as a type 5 route.

  7. 7What the spines needSPINE1
    SPINE1 on NX-OS
    nv overlay evpn
!
route-map RM-NH-UNCHANGED permit 10
  set ip next-hop unchanged
!
router bgp 65100
  address-family l2vpn evpn
    retain route-target all
  neighbor 10.0.0.11
    remote-as 65001
    update-source loopback0
    ebgp-multihop 2
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map RM-NH-UNCHANGED out
  neighbor 10.0.0.12
    remote-as 65002
    update-source loopback0
    ebgp-multihop 2
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map RM-NH-UNCHANGED out

    The spines have no VNIs, so without retain route-target all they would drop every EVPN route. The next hop must stay the originating leaf's VTEP; eBGP would otherwise rewrite it to the spine. SPINE2 is the same.

Verify

LEAF1 output
LEAF1# show nve peers
Interface Peer-IP                                 State LearnType Uptime   Router-Mac
--------- --------------------------------------  ----- --------- -------- -----------------
nve1      10.0.1.12                               Up    CP        00:06:18 a4b2.3912.0c1b

LEAF2's VTEP is a peer, learned through the control plane (CP), with its router MAC for symmetric IRB.

LEAF1 output
LEAF1# show nve vni
Codes: CP - Control Plane        DP - Data Plane
       UC - Unconfigured         SA - Suppress ARP
       SU - Suppress Unknown Unicast
       Xconn - Crossconnect
       MS-IR - Multisite Ingress Replication

Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1      10010    UnicastBGP        Up    CP   L2 [10]            SA
nve1      50001    n/a               Up    CP   L3 [TENANT-A]

The L2VNI uses BGP ingress replication with ARP suppression (SA). The L3VNI is up and bound to the VRF.

LEAF1 output
LEAF1# show l2route evpn mac all
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending
(S):Stale (C):Clear, (Ps):Peer Sync (O):Re-Originated (Nho):NH-Override
(Pf):Permanently-Frozen, (Orp): Orphan

Topology    Mac Address    Prod   Flags         Seq No     Next-Hops
----------- -------------- ------ ------------- ---------- ----------------
10          0050.56a1.0011 Local  L,            0          Eth1/1
10          0050.56a1.0012 BGP    Rcv           0          10.0.1.12 (Label: 10010)

SRV1's MAC is local on Eth1/1. SRV2's MAC was learned from a BGP type 2 route and points at LEAF2's VTEP.

LEAF1 output
LEAF1# show ip route vrf TENANT-A 10.10.20.22
IP Route Table for VRF "TENANT-A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.10.20.22/32, ubest/mbest: 1/0
    *via 10.0.1.12%default, [20/0], 00:05:41, bgp-65001, external, tag 65100 (evpn) segid: 50001 tunnelid: 0xa00010c encap: VXLAN

LEAF1 has no VLAN 20, yet it has a host route to SRV3 from LEAF2's MAC+IP route, through VNI 50001 toward 10.0.1.12 (0xa00010c in hex). That is symmetric IRB.

SRV1 output
SRV1# ping -c 3 10.10.10.12
PING 10.10.10.12 (10.10.10.12) 56(84) bytes of data.
64 bytes from 10.10.10.12: icmp_seq=1 ttl=64 time=0.412 ms
64 bytes from 10.10.10.12: icmp_seq=2 ttl=64 time=0.388 ms
64 bytes from 10.10.10.12: icmp_seq=3 ttl=64 time=0.391 ms

--- 10.10.10.12 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2031ms
rtt min/avg/max/mdev = 0.388/0.397/0.412/0.011 ms

TTL 64 (the Linux default, untouched) shows SRV1 and SRV2 are bridged in the same VNI, not routed.

Mistakes people make

  • Using route-target auto with one ASN per leaf. The derived values differ per leaf and nothing is imported.
  • Forgetting send-community extended on either the leaf or the spine. Routes arrive without route targets.
  • Spines that rewrite the next hop or lack retain route-target all. Routes vanish or point at a spine.
  • Missing or shut L3VNI VLAN or SVI. VNI 50001 stays down and routing between subnets fails while bridging works.
  • Different anycast gateway MACs on different leaves. Hosts that move keep ARPing for a MAC that no longer answers.

On other platforms

The same leaf on Arista EOS. No VLAN is needed for the L3VNI, ip address virtual gives the anycast gateway, and the EVPN instance lives under router bgp. Older EOS releases also need service routing protocols model multi-agent and a reload before EVPN works.

Arista EOS
ip virtual-router mac-address 20:20:00:00:00:aa
!
vlan 10
   name WEB
vrf instance TENANT-A
ip routing vrf TENANT-A
!
interface Ethernet1
   switchport access vlan 10
   spanning-tree portfast
interface Vlan10
   vrf TENANT-A
   ip address virtual 10.10.10.1/24
interface Vxlan1
   vxlan source-interface Loopback1
   vxlan udp-port 4789
   vxlan vlan 10 vni 10010
   vxlan vrf TENANT-A vni 50001
!
router bgp 65001
   neighbor SPINE-EVPN peer group
   neighbor SPINE-EVPN remote-as 65100
   neighbor SPINE-EVPN update-source Loopback0
   neighbor SPINE-EVPN ebgp-multihop 2
   neighbor SPINE-EVPN send-community extended
   neighbor 10.0.0.1 peer group SPINE-EVPN
   neighbor 10.0.0.2 peer group SPINE-EVPN
   !
   vlan 10
      rd 10.0.0.11:10010
      route-target both 65000:10010
      redistribute learned
   !
   address-family evpn
      neighbor SPINE-EVPN activate
   !
   address-family ipv4
      no neighbor SPINE-EVPN activate
   !
   vrf TENANT-A
      rd 10.0.0.11:50001
      route-target import evpn 65000:50001
      route-target export evpn 65000:50001
      redistribute connected

Say it in the interview

"VLAN 10 maps to VNI 10010 and VLAN 3001 to the L3VNI 50001 for VRF TENANT-A. The SVI uses the fabric-wide anycast gateway MAC. NVE sources from loopback1, uses BGP for host reachability with ingress replication and ARP suppression. Because every leaf has its own ASN, I set route targets by hand. The EVPN sessions run loopback to loopback, and the spines retain all route targets and leave the next hop alone. I verify NVE peers, VNI state, MACs learned from BGP and a host route through the L3VNI."

Book Perspective: Virtual Networks All the Way Down