Data center

Three tier

• Access switches connect servers, aggregation holds the SVIs, the FHRP and often the services (firewalls, load balancers), core routes between aggregation blocks and to the WAN.
• VLANs stretch from access to aggregation, so STP is in charge of loop prevention and blocks redundant uplinks. Half the bandwidth you paid for sits idle.
• Server-to-server traffic in different blocks climbs to the core and back down. Latency and available bandwidth depend on where the two servers sit.
• Oversubscription compounds at every tier, and growth means buying a bigger aggregation chassis (scale up).

Spine-leaf

• Two layers. Servers, firewalls, routers and storage connect only to leaves. Spines only connect leaves.
• Leaf to spine links are routed, usually /31 point-to-point. No STP between switches.
• Every leaf has an equal-cost path through every spine, so ECMP uses all uplinks.
• Any two servers on different leaves are the same distance apart. Latency is predictable.
• Losing one spine out of four removes 25% of fabric bandwidth, not half of a block.
• Layer 2 between racks, when you need it, is rebuilt on top as an overlay (VXLAN with BGP EVPN).

What shifted the traffic

• Multi-tier and microservice apps. One user request fans out into many internal calls.
• Distributed storage and databases. Replication and rebuilds move large volumes between racks.
• Virtualization. Live migration copies VM memory across the network.
• Big data and AI training. Shuffle phases and collective operations hit every node at once.

What it changed

• Topology. Three tier sends cross-block east-west traffic up through aggregation and core. Spine-leaf makes every rack two hops from every other rack.
• Capacity planning. You size leaf uplinks (oversubscription) for server-to-server load, not just for the internet link.
• Security. A perimeter firewall only sees north-south. East-west needs its own controls: segmentation, inter-VRF firewalls, microsegmentation.
• Layer 2 scope. Apps wanted the same subnet in many racks, which pushed overlays such as VXLAN.

The math

What the ratio really means

• It is the worst case: every server sending at line rate toward other racks at the same moment. Real traffic is burstier and rarely all leaves the rack.
• In a two-stage fabric the spines are not oversubscribed, so the leaf ratio is the fabric ratio. In a five-stage fabric the pod spine to super-spine ratio multiplies with it.
• Plan for failure. With 6 uplinks and one spine down, 1,200G over 500G is 2.4:1.
• Count only the ports you will really use. A rack with 30 servers on a 48-port leaf has a better effective ratio than the box suggests.

Underlay

• Point-to-point routed links between leaves and spines, plus loopbacks (router ID and VTEP address).
• Tiny routing table: a few hundred loopbacks even in a large fabric.
• Built for fast convergence and ECMP. MTU set large (9216 is common) to carry VXLAN overhead.

Overlay

• VXLAN encapsulation with a VNI per layer 2 segment and a VNI per tenant VRF.
• BGP EVPN advertises host MACs, host IPs, prefixes and which VTEPs belong to which VNI.
• Holds the scale: thousands of MACs and host routes, many tenants.

Why separate them

• Failure isolation: a tenant change cannot break fabric reachability.
• Simple underlay, rich overlay: you can rebuild tenants without touching the physical design.

Troubleshooting order

Underlay first. Ping VTEP loopback to VTEP loopback, sourced from the loopback, with a full-size packet and DF set. If that fails, nothing above it will work. Then check the EVPN control plane, then the data plane.

Practical points

• At 25G and 100G, passive DAC works only for a few meters, which favors top-of-rack. Longer runs need active cables or optics.
• Top-of-rack pairs (MLAG or vPC) let you dual-home every server inside the rack.
• Middle-of-row shortens the longest cable run compared with end-of-row.

The problems it answers

• Scale of segments. 4,094 VLANs is not enough for a multi-tenant or cloud data center.
• Spanning tree. Stretching VLANs across many racks the classic way means STP, blocked links and large failure domains.
• Workload mobility. VMs and containers want to keep their IP when they move to another rack.
• MAC table size in the core. Underlay switches only see VTEP addresses, not every host MAC.

How it works

A VTEP on the leaf maps a local VLAN to a VNI, wraps the frame in VXLAN, UDP and IP, and sends it to the remote VTEP's loopback. The underlay forwards it like any other UDP packet.

Related encapsulations

GENEVE (UDP 6081) has the same 24-bit VNI plus variable-length options. VMware NSX and some cloud services use it. NVGRE and STT were alternatives that did not win in physical fabrics.

On each platform

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp

interface Vxlan1
   vxlan source-interface Loopback1
   vxlan udp-port 4789
   vxlan vlan 10 vni 10010

ip link add vxlan10010 type vxlan id 10010 local 10.0.1.11 dstport 4789 nolearning

Why a separate VTEP loopback

Most designs use loopback0 for the router ID and BGP peering and loopback1 for the VTEP. In a vPC or MLAG pair, both switches share one VTEP address (an anycast VTEP; on NX-OS a secondary IP on loopback1) so remote leaves see one tunnel endpoint for dual-homed hosts. The router IDs stay unique.

What the VTEP knows

• Local MACs, learned from frames on access ports.
• Remote MACs and IPs, learned from EVPN type 2 routes (or from decapsulated traffic in flood and learn mode).
• Peers, the remote VTEPs it has tunnels to: show nve peers on NX-OS, show vxlan vtep on EOS.

Configuration

feature fabric forwarding
fabric forwarding anycast-gateway-mac 2020.0000.00aa
!
interface Vlan10
  vrf member TENANT-A
  ip address 10.10.10.1/24
  fabric forwarding mode anycast-gateway

ip virtual-router mac-address 00:1c:73:00:00:99
!
interface Vlan10
   vrf TENANT-A
   ip address virtual 10.10.10.1/24

The gateway MAC is one value for the whole fabric, configured identically on every leaf.

Compared with HSRP or VRRP

• HSRP has one active router per group. In a fabric, every rack's traffic would cross to that router.
• FHRPs exchange hellos and fail over in seconds. An anycast gateway has nothing to fail over.
• A moved VM keeps the same gateway MAC and IP, so it does not need to re-ARP.

Operational points

• The gateway IP is shared, so you cannot use it to reach one specific leaf. Give each leaf a unique loopback in the tenant VRF for monitoring.
• DHCP relay must use a unique source per leaf (that loopback) so the server's reply returns to the right leaf, with option 82 link selection telling the server which scope to use.

Building blocks

How the pair looks like one switch

• Both peers send the same LACP system ID. NX-OS derives a vPC system MAC from the domain ID (0023.04ee.bexx, where xx is the domain ID in hex).
• MAC addresses, ARP entries (with ip arp synchronize) and IGMP snooping state are synced across the peer link.
• Each peer forwards locally. The peer link carries data only for single-attached devices and failures.
• Loop prevention: a frame that arrived over the peer link is not sent out a vPC member port, unless the peer's matching member port is down.

Compared with stacking or VSS

Stacking, VSS and StackWise Virtual merge two switches into one control plane with one config. vPC and MLAG keep two brains. You upgrade one peer at a time, but you must keep the configs consistent, which is what the consistency checks enforce.

Layer 4 modes

• NAT: rewrites the destination VIP to a server address, one TCP connection end to end.
• Full proxy at layer 4: separate client-side and server-side TCP connections, no payload parsing.
• Direct server return: only the request passes the load balancer.

Layer 7

Always a full proxy with two connections. It can reuse server-side connections, speak HTTP/2 to clients and HTTP/1.1 to servers, and make per-request decisions on the same client connection.

TLS choices

Same idea on different products

HAProxy mode tcp vs mode http, F5 FastL4 vs a standard virtual server with an HTTP profile, AWS NLB vs ALB, NGINX stream vs http.

Persistence

• Source IP affinity: breaks behind carrier NAT and for mobile users who change IP.
• Cookie insert: the layer 7 balancer sets a cookie naming the server. The most reliable for HTTP.

Defaults

HAProxy uses balance roundrobin unless told otherwise. F5 pools default to round robin. Envoy's least request picks two random hosts and sends to the less busy one.

Probe types

Timers you should know

• HAProxy defaults: inter 2s, fall 3, rise 2. A dead server is removed after about 6 seconds.
• F5 HTTP monitor defaults: interval 5 seconds, timeout 16 seconds.

Design rules

• Shallow, not deep. If every server's check queries the same database, a database hiccup removes the whole pool and turns a slowdown into an outage.
• Drain gracefully. Fail the check or set the member to drain before maintenance so existing sessions finish.
• Know the all-down behavior. Some balancers return 503, others fail open. AWS ALB, for example, sends traffic to all targets when every target is unhealthy.
• Probe source. Checks come from the load balancer's own address. Firewalls and host ACLs must allow it, and a green check does not prove that client traffic works.

Block and file

A SAN gives hosts raw block devices (FC, iSCSI, FCoE, NVMe-oF). A NAS shares files (NFS, SMB).

Fibre Channel essentials

• WWNN and WWPN identify nodes and ports, like burned-in MACs. The switch assigns a 24-bit FCID at fabric login.
• Zoning on the switches decides which initiators may talk to which targets. LUN masking on the array decides which volumes a host sees.
• Credits. A port only transmits when the receiver has advertised buffer space, so FC does not drop for congestion. A slow-drain device instead backs up the fabric.
• Two fabrics. Fabric A and fabric B are physically separate and never joined. Hosts use multipathing across them. Cisco MDS adds VSANs to split one physical fabric into logical ones.

iSCSI essentials

• Initiator (host) and target (array), named by IQNs such as iqn.2024-01.com.example:array1.
• TCP 3260, CHAP for authentication, no encryption by default.
• Multipathing across two subnets rather than link aggregation.

FCoE and DCB

FCoE needs PFC (802.1Qbb) for a no-drop class, ETS (802.1Qaz) for bandwidth shares and DCBX to negotiate them. FIP (Ethertype 0x8914) handles discovery and login. It saw use at the server edge and has faded.

NVMe over Fabrics

NVMe/TCP (port 4420) runs on ordinary networks. NVMe over RoCEv2 (UDP 4791) needs a lossless class.

Worked example

Leaves with 48 x 25G down and 6 x 100G up, spines with 64 x 100G:

• Up to 6 spines (one uplink from each leaf to each spine).
• Up to 64 leaves (one spine port per leaf), minus any ports used for border leaves.
• 64 leaves x 48 ports = 3,072 server ports at 25G, or 1,536 dual-homed servers.

How it grows

• More ports: add a leaf (or leaf pair) and cable it to every spine. No change to existing racks.
• More bandwidth: add a spine and one uplink from every leaf. Each leaf gains a new ECMP path.
• Past one spine's port count: build pods (each a spine-leaf), then connect pod spines to a super-spine layer. That is a five-stage Clos. Traffic between pods takes four hops.

Roles on the leaves

Not every leaf has servers. A border leaf connects WAN routers or the internet edge, and a service leaf connects firewalls and load balancers. They are still leaves and follow the same rules.

Making it work

• BGP installs one best path by default on IOS, NX-OS and EOS. Configure maximum-paths (eBGP) to at least the number of spines. On EOS it is maximum-paths 4 ecmp 4.
• With OSPF or IS-IS, equal-cost paths are installed automatically up to the platform's limit.
• VXLAN keeps ECMP useful: the VTEP sets the outer UDP source port from a hash of the inner headers, so different inner flows between the same two VTEPs take different spines.

What goes wrong

• Elephant flows. Backups, storage rebuilds and replication are few and huge. Two of them can land on the same 100G uplink while others idle. Fixes: more or faster uplinks, more parallel streams, or dynamic load balancing (flowlet switching) where the ASIC supports it.
• Polarization. If every switch hashes the same fields with the same function, flows that chose link 1 at one stage all make the same choice at the next stage. Vendors add a per-switch seed.
• Rehash on change. Adding or removing a path can move many flows. Resilient hashing moves only the flows that used the failed path.
• Unequal paths. One leaf with two links to a spine and one link to another breaks the "equal" in ECMP. Keep cabling symmetric.

Arguments for eBGP

• No flooding. A link flap in a link-state IGP floods LSAs everywhere and triggers SPF on every switch. In eBGP only the affected paths are updated.
• Policy. Route-maps and prefix lists per neighbor. You can drain a spine by prepending AS path.
• One protocol. The same BGP stack carries L2VPN EVPN for the overlay.
• Loop prevention by ASN. With all spines in one ASN, a route cannot go leaf, spine, leaf, spine.

What you must tune

• Detection. Default BGP timers are 60 s keepalive and 180 s hold. Use BFD, and rely on immediate session teardown when a directly connected link goes down.
• Multipath. maximum-paths must be raised or there is no ECMP.
• ASN plan. See the ASN scheme question. Bad plans cause path hunting or dropped routes.

When I would pick OSPF or IS-IS

A small or medium fabric, a team that knows OSPF well, or a vendor design that expects an IGP underlay with spines acting as iBGP route reflectors for EVPN. Both approaches work. Consistency matters more.

Failure detection

• Link down. Fastest. Directly connected eBGP sessions drop immediately when the interface goes down.
• BFD. Catches failures where light stays on: a hung line card, a one-way fiber, a DWDM or media converter in the middle. Typical values are a few hundred milliseconds total detection.
• BGP hold timer. 180 s by default. Never rely on it in a fabric.

Clean drain options

• AS-path prepend outbound on the spine toward every leaf. Leaves still have the path as a backup.
• GRACEFUL_SHUTDOWN community 65535:0 (RFC 8326). Receivers that match it set local preference to 0.
• Vendor maintenance mode. NX-OS Graceful Insertion and Removal (system mode maintenance) isolates the switch from routing protocols. EOS has its own maintenance mode for BGP peers and interfaces.

Upgrade routine

1. Check the remaining spines can carry the load (oversubscription with N minus one).
2. Drain, then confirm interface counters on the spine drop to near zero.
3. Upgrade and reload, verify sessions and routes, then undrain.

Layout

+----------------+------------+---------+----------+---------------------------------+
| Outer Ethernet | Outer IPv4 | UDP     | VXLAN    | Inner Ethernet frame            |
| 14 bytes       | 20 bytes   | 8 bytes | 8 bytes  | dst MAC, src MAC, type, payload |
+----------------+------------+---------+----------+---------------------------------+

The 8-byte VXLAN header

| Flags (8 bits, I flag set) | Reserved (24 bits)            |
| VNI (24 bits)                              | Reserved (8)  |

The I flag set to 1 means the VNI field is valid. Everything else is reserved and zero.

Outer header details

• Outer IP source is the local VTEP loopback. Destination is the remote VTEP, or a multicast group for BUM traffic in multicast mode.
• UDP destination 4789 is the IANA port. The Linux kernel defaults to 8472 (the pre-standard port) unless you create the interface with dstport 4789.
• UDP source port comes from a hash of the inner frame. RFC 7348 recommends the 49152 to 65535 range.
• No fragmentation. VTEPs must not fragment VXLAN packets, so the underlay MTU has to fit them.
• The inner 802.1Q tag is normally removed: the VLAN is mapped to the VNI and recreated at the far end.

Overhead variations

The math

Inner IP packet 1500 + inner Ethernet header 14 = 1514. Add VXLAN 8, UDP 8 and outer IPv4 20 and the outer IP packet is 1550 bytes. The underlay interface must carry at least that, plus the outer Ethernet header on the wire.

Proving it from a Linux host

ping -M do -s 1472 10.10.10.12    # 1472 + 28 = 1500-byte packet, fails if the underlay is at 1500
ping -M do -s 1422 10.10.10.12    # 1450-byte packet, passes

Then test the underlay itself from the leaf: ping the remote VTEP loopback, sourced from the local VTEP loopback, with the DF bit set and a large size (NX-OS ping 10.0.1.12 source-interface loopback1 df-bit packet-size 8000). Walk the path link by link with show interface | include MTU to find the odd one.

Why PMTUD does not save you

The underlay switch that drops the packet sends ICMP "fragmentation needed" to the outer source, which is the VTEP, not the host. VTEPs do not translate it back to the inner sender, so the host keeps sending 1500-byte packets into a black hole.

The fix

• Every fabric link (leaf to spine, DCI, border) at a jumbo MTU, the same everywhere.
• Or, where the underlay cannot be raised (some WAN or cloud paths), lower the host MTU to 1450.

Points worth saying out loud

• EVPN is the control plane, VXLAN is the data plane. EVPN also runs over MPLS.
• EVPN still learns local MACs from the data plane on access ports, then advertises them.
• EVPN is MP-BGP with AFI 25 (L2VPN) and SAFI 70 (EVPN).
• Flood and learn trusts any VTEP that sends traffic. EVPN only installs what BGP advertised.

Multicast underlay

• VTEPs join the group for each local VNI. The outer destination IP of BUM packets is the group.
• Needs PIM sparse mode (often anycast RP on the spines) or PIM BiDir.
• Efficient for many VTEPs and heavy BUM, but adds multicast state and a new class of problems.

interface nve1
  member vni 10010
    mcast-group 239.1.1.10

Ingress replication

• EVPN type 3 routes tell every VTEP who else is in the VNI, and the PMSI tunnel attribute says ingress replication. NX-OS: ingress-replication protocol bgp. EOS builds the flood list from type 3 automatically.
• Flood and learn can use a static list (ingress-replication protocol static with peer addresses).
• Cost example: 40 VTEPs in a VNI and 2 Mbps of broadcast means 78 Mbps leaving the source leaf. Usually small, because ARP suppression removes most broadcasts.

Types 6 to 8 handle IGMP and MLD proxy for selective multicast.

What they look like on NX-OS

[2]:[0]:[0]:[48]:[0050.56a1.0011]:[0]:[0.0.0.0]/216        MAC only
[2]:[0]:[0]:[48]:[0050.56a1.0011]:[32]:[10.10.10.11]/272   MAC and IP
[3]:[0]:[32]:[10.0.1.11]/88                                VTEP 10.0.1.11 is in this VNI
[5]:[0]:[0]:[24]:[10.10.20.0]/224                          prefix 10.10.20.0/24

In a type 2 string the fields are type, ESI, Ethernet tag, MAC length, MAC, IP length, IP.

Standards

RFC 7432 (EVPN base), RFC 8365 (EVPN with VXLAN), RFC 9135 (integrated routing and bridging) and RFC 9136 (type 5 prefixes).

Fields, with example values

Two routes per host

Leaves often send a MAC-only route and a MAC+IP route for each host. The first is enough for bridging. The second drives routing and ARP suppression.

What the receiver installs

• MAC table: 0050.56a1.0011 in VLAN 10, next hop VTEP 10.0.1.11, VNI 10010.
• VRF table: 10.10.10.11/32 via 10.0.1.11, VNI 50001, inner destination MAC set to that leaf's router MAC.
• ARP suppression cache: 10.10.10.11 is at 0050.56a1.0011.

Typical uses

• Border leaf. Learns 0.0.0.0/0 from a firewall or WAN router over eBGP in the VRF (VRF-lite handoff) and re-advertises it as type 5 to every leaf.
• Subnet routes. Each leaf advertises its SVI subnets. A leaf that does not have VLAN 20 routes toward 10.10.20.0/24 using the type 5, and the receiving leaf ARPs for the host.
• Silent hosts. A host that never sends has no type 2. The type 5 gets traffic to a leaf that owns the subnet, which then resolves the host.
• Layer 3 only tenants and DCI, including summaries between sites.

Config sketch

router bgp 65001
  vrf TENANT-A
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map RM-TENANT-SUBNETS

router bgp 65001
   vrf TENANT-A
      rd 10.0.0.11:50001
      route-target import evpn 65000:50001
      route-target export evpn 65000:50001
      redistribute connected

Type 2 and type 5 together

A /32 host route from a type 2 is more specific than the /24 type 5, so longest match sends traffic straight to the leaf where the host really is. The type 5 is the fallback.

Flow

1. SRV2 on LEAF2 sends its first packet. LEAF2 learns its MAC and IP and advertises a type 2 MAC+IP route.
2. LEAF1 installs 10.10.10.12 at 0050.56a1.0012 in its suppression cache.
3. SRV1 on LEAF1 ARPs for 10.10.10.12. LEAF1 answers locally. Nothing crosses the fabric.

Config and checks

interface nve1
  member vni 10010
    suppress-arp

show ip arp suppression-cache detail lists local (L) and remote (R) entries. On some older Nexus 9000 hardware ARP suppression needs a TCAM region carved first, which takes a reload. IPv6 has the same idea for neighbor discovery.

Limits

• A silent host has no entry, so the first ARP for it is flooded. That is normal.
• DHCP and other broadcasts are untouched.

Failure table

Roles

Primary and secondary are decided by role priority (lower wins, default 32667), then by system MAC. The election is not preemptive, so after a reload the roles may stay swapped.

Design rules

• Peer link: at least two members on different line cards or ASICs.
• Keepalive: never over the peer link. Use mgmt0 in the management VRF, or a dedicated routed link in its own VRF.
• auto-recovery: lets the secondary take over when it concludes the primary is really gone, and lets a switch that boots alone bring its vPCs up after a delay (240 seconds by default).

Arista MLAG difference

Arista sends MLAG heartbeats over the peer link itself, on a dedicated VLAN (4094 by convention). For split-brain protection you add peer-address heartbeat over the management network and a dual-primary detection action.

In normal operation

Orphan traffic often has to cross the peer link to reach devices that hash to the other peer. Size the peer link for it.

Finding them

show vpc orphan-ports lists every orphan port per VLAN.

Fixing them

interface Ethernet1/10
  description SRV2 nic1, active-standby team
  vpc orphan-port suspend

Use it on physical ports facing hosts that can fail over to a second NIC on the other peer. For devices with one NIC, the real fix is a second link.

SNAT port exhaustion

Every connection to the same server IP and port needs a unique SNAT address and port pair, so one SNAT address supports roughly 64,000 concurrent connections per server and port. Busy VIPs use a SNAT pool of several addresses. The symptom is intermittent connection failures under load.

Direct server return on Linux servers

ip addr add 10.20.0.100/32 dev lo
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2

The VIP on loopback lets the server accept packets for it. The two sysctls stop the server answering ARP for the VIP so only the load balancer owns it on the segment. The load balancer rewrites only the destination MAC, so servers must share the layer 2 segment, unless you use IP-in-IP tunneling as Linux IPVS tunnel mode does.

Restoring the client IP with SNAT

• HTTP: X-Forwarded-For (HAProxy option forwardfor, an F5 HTTP profile setting).
• Any TCP: PROXY protocol prepended to the connection. The server must be configured to expect it.

Layout

Why MPIO and not LACP

LACP hashes one iSCSI session onto one link, so it adds no per-session throughput and fails over at the link layer without knowing the array. MPIO sees every path, balances I/O across them and fails over with awareness of the array's controllers. Most storage vendors require MPIO for iSCSI.

MTU

9000 on hosts and array, switch ports at 9216 or the platform maximum. Test each host NIC with ping -M do -s 8972 <target>. A mismatch shows as sessions that log in fine and then stall on large reads.

Other points

• Avoid routing and firewall inspection between initiator and target. Both add latency and session drops.
• Edge ports (PortFast) on host and array ports so a topology change does not stall storage.
• Follow the array vendor on Ethernet flow control. Recommendations differ.
• Dedicated switches, or QoS when shared. Many hosts reading at once cause microbursts.

Transport

Dark fiber or DWDM (lowest latency, you own the optics), carrier MPLS or Metro Ethernet, and internet IPsec as a backup.

Layer 3 only

Each site has its own subnets. BGP between sites. Service failover through GSLB, DNS or anycast. Replication in the database or application. The cleanest failure isolation.

Layer 2 extension options

Risks of stretched layer 2

• Shared failure domain. A loop in one site floods the other.
• Tromboning. A host in site B uses a gateway in site A. Fix with an anycast gateway in both sites.
• Ingress asymmetry. Traffic enters through site A's firewall and leaves through site B's, and stateful firewalls drop it.
• Split brain. Clusters need a quorum witness in a third location.
• Latency. Synchronous replication and live migration have distance limits.

Enforcement points

Rolling it out

1. Get visibility: flow telemetry or host agents to map who talks to whom.
2. Define groups by labels, such as app=payroll, tier=db, env=prod.
3. Write allow rules between groups and run them in log-only mode.
4. Enforce default deny one application at a time.
5. Keep policy in version control with review.

Why the spines share an ASN

When LEAF2's loopback is withdrawn, a spine with its own ASN could pick a backup path through another leaf and another spine, then withdraw that too. That is path hunting and it slows convergence. When all spines are 65100, any path that would cross a second spine contains 65100 twice and is rejected on arrival.

Why each leaf gets its own ASN

• If all leaves were 65001, LEAF1 would drop routes from LEAF2 because 65001 is already in the path. You would need allowas-in on leaves, which weakens loop protection.
• LEAF1 receives LEAF2's prefix from SPINE1 and SPINE2 with the same path 65100 65002, so plain maximum-paths installs both.

Leaf pairs

MLAG or vPC peers commonly share one ASN and run an iBGP session across the peer link, so either peer can still reach the fabric if it loses all its uplinks.

ASN ranges

2-byte private ASNs are 64512 to 65534 (1,023 numbers). Large fabrics use 4-byte private ASNs, 4200000000 to 4294967294 (RFC 6996).

When you do need multipath-relax

When the same prefix arrives with different AS paths of equal length, for example an anycast service advertised from leaves in 65001 and 65002 (65100 65001 and 65100 65002). IOS and NX-OS only use both paths with bestpath as-path multipath-relax (IOS spells it bgp bestpath as-path multipath-relax).

Why it hurts so much

• Data center RTTs are tens to hundreds of microseconds. A 200 ms RTO stall is a thousand times longer.
• Losses are synchronized, so many flows stall together and the link sits idle.
• Five-minute or even one-minute interface counters average it away. These are microbursts.

Network side

• Buffers. Shallow-buffer switches share a few tens of MB across all ports. Deep-buffer platforms at congestion points (storage leaves, border, DCI) absorb bursts. Tune the shared-buffer thresholds.
• ECN. Switches mark packets instead of dropping when a queue passes a threshold.
• Telemetry. Watch per-queue drops and buffer occupancy, not interface averages.

Host side

• DCTCP reacts to the fraction of ECN-marked packets with a proportional slowdown, keeping queues short.
• Lower RTO minimum on Linux per route (ip route change ... rto_min 5ms) in controlled environments.
• Application pacing. Add jitter or limit how many servers answer at once.

RDMA is different

RoCEv2 (UDP 4791) does not tolerate loss well, so it runs on lossless classes with PFC (802.1Qbb) plus ECN.

Route distinguisher

• NX-OS rd auto for a MAC-VRF is router-ID:(32767 + VLAN ID). VLAN 10 on router ID 10.0.0.11 gives 10.0.0.11:32777.
• Keep RDs unique per leaf. If two leaves used the same RD for the same prefix, a spine or route reflector would see one NLRI, keep one best path and hide the other leaf, which breaks ECMP to anycast or multihomed destinations.

Route target

• Set per MAC-VRF (L2VNI) and per IP-VRF (L3VNI). A MAC+IP type 2 carries both.
• Import policy can differ from export: a shared-services VRF can import from many tenants.

The eBGP auto RT mismatch

Fixes:

1. Manual RTs with a fabric convention, for example route-target both 65000:10010 for VNI 10010.
2. rewrite-evpn-rt-asn under the L2VPN EVPN address family of each eBGP neighbor on NX-OS: the receiver replaces the ASN part of incoming RTs with its own before import.

Spine behavior with eBGP EVPN

• NX-OS spines have no VNIs, so they drop routes with no matching import RT unless you configure retain route-target all under address-family l2vpn evpn.
• Spines must keep the next hop: NX-OS uses an outbound route-map with set ip next-hop unchanged, EOS uses neighbor ... next-hop-unchanged.

Example

SRV1 10.10.10.11 in VLAN 10 on LEAF1 talks to SRV3 10.10.20.22 in VLAN 20 on LEAF2. VRF TENANT-A, L2VNIs 10010 and 10020, L3VNI 50001.

Asymmetric

1. LEAF1 routes from VLAN 10 to VLAN 20 using its own VLAN 20 SVI.
2. LEAF1 bridges to SRV3's MAC in VNI 10020.
3. LEAF2 decapsulates and bridges to SRV3. The reply is routed by LEAF2 and sent in VNI 10010.

Symmetric

1. LEAF1 looks up 10.10.20.22/32 in VRF TENANT-A, learned from LEAF2's type 2 MAC+IP route.
2. LEAF1 encapsulates in VNI 50001. Inner source MAC is LEAF1's router MAC, inner destination MAC is LEAF2's router MAC.
3. LEAF2 maps VNI 50001 to the VRF, routes to VLAN 20 and delivers to SRV3. The reply uses VNI 50001 too.

Sequence numbers

• The first advertisement of a MAC carries no mobility community, which counts as sequence 0.
• Each move adds one. The highest sequence wins, before normal BGP best path comparison matters.
• If two leaves claim the same sequence number, RFC 7432 picks the route from the lower originator IP.
• A MAC advertised as static (sticky) cannot move. Gateway MACs use this.

With an anycast gateway

The VM keeps its cached gateway ARP entry because every leaf owns the same gateway MAC. Its /32 host route moves with the MAC+IP type 2 route, so routed traffic follows it.

Duplicate detection

• Real causes: a layer 2 loop at the access layer, cloned VMs with the same MAC, a broken multihoming setup that learns the same MAC on both sides.
• NX-OS marks the entry with a Dup flag in show l2route evpn mac all and stops advertising it. After repeated detections it can freeze the MAC until you clear it.
• Duplicate IP detection works the same way for MAC+IP routes.

What each piece does

• Type 4 Ethernet segment route. Leaves with the same ESI find each other and elect a designated forwarder (DF) per VLAN, so only one leaf sends BUM traffic down to the host.
• Type 1 per segment. If a leaf loses its link, one withdrawal tells every remote VTEP to stop using that leaf for all MACs on the segment (mass withdraw).
• Type 1 per EVI. Aliasing: remote leaves load-balance to every leaf on the segment, even if only one of them learned the MAC.
• Split horizon. With VXLAN, local bias: BUM from a host is delivered to the segment by the leaf it arrived on, and peers on the same segment do not send it back.

Config on EOS

interface Port-Channel10
   switchport mode trunk
   switchport trunk allowed vlan 10
   evpn ethernet-segment
      identifier 0000:0000:0102:0304:0506
      route-target import 01:02:03:04:05:06
   lacp system-id 0102.0304.0506

The same block, with the same ESI and system ID, goes on the other leaf.

The VRF sandwich

VRF PROD on the fabric  -> VLAN 3901 -> FW1 inside-prod interface
VRF DEV on the fabric   -> VLAN 3902 -> FW1 inside-dev interface

Each VRF learns 0.0.0.0/0 from FW1. The service leaves export it as a type 5 route with that VRF's L3VNI. A PROD host sending to a DEV host follows the default route to the service leaf, crosses the firewall and enters VRF DEV.

What bites

• Symmetry. A stateful firewall must see both directions. Active/active firewalls on different leaves or sites need shared state or pinned flows.
• Failover. An active/standby pair moves its MAC and IP. EVPN MAC mobility handles it, but attach each firewall with vPC, MLAG or ESI so one link flap does not cause a move.
• Capacity. All inter-VRF traffic hairpins to the service leaves. Size the firewall and those uplinks.
• Route leaking. Leaking routes between VRFs on the fabric for shared services bypasses the firewall. Do it on purpose and narrowly.
• Intra-VRF traffic. Never reaches this firewall, because the ingress leaf routes it locally. Use policy-based redirection (NX-OS ePBR, ACI service graphs) or microsegmentation for that.