What is the difference between the underlay and the overlay in a data center fabric?
Show the answer
Say this first
The underlay is the physical routed network between the switches. Its only job is IP reachability between switch loopbacks, with ECMP, using eBGP, OSPF or IS-IS. The overlay is the set of virtual networks built on top of it: VXLAN tunnels between VTEPs carrying tenant layer 2 and layer 3 traffic, with BGP EVPN as the control plane. The underlay knows nothing about tenants, VLANs or host MACs.
Go deeper
Underlay
- Point-to-point routed links between leaves and spines, plus loopbacks (router ID and VTEP address).
- Tiny routing table: a few hundred loopbacks even in a large fabric.
- Built for fast convergence and ECMP. MTU set large (9216 is common) to carry VXLAN overhead.
Overlay
- VXLAN encapsulation with a VNI per layer 2 segment and a VNI per tenant VRF.
- BGP EVPN advertises host MACs, host IPs, prefixes and which VTEPs belong to which VNI.
- Holds the scale: thousands of MACs and host routes, many tenants.
Why separate them
- Failure isolation: a tenant change cannot break fabric reachability.
- Simple underlay, rich overlay: you can rebuild tenants without touching the physical design.
Troubleshooting order
Underlay first. Ping VTEP loopback to VTEP loopback, sourced from the loopback, with a full-size packet and DF set. If that fails, nothing above it will work. Then check the EVPN control plane, then the data plane.
They will ask next
Can underlay and overlay both run BGP?
Yes, and often do. eBGP IPv4 unicast on the physical links for the underlay, plus eBGP L2VPN EVPN sessions between loopbacks for the overlay. Some designs carry both address families on the same sessions.
Traps
- Putting tenant subnets into the underlay routing table. They belong in VRFs in the overlay.