Walk me through a VXLAN packet. Which headers are added, and how much overhead is it?
Show the answer
Say this first
The VTEP adds an outer Ethernet header (14 bytes), an outer IPv4 header (20), a UDP header (8) with destination port 4789, and the 8-byte VXLAN header holding a 24-bit VNI. That is 50 bytes on an IPv4 underlay without an outer VLAN tag. The original frame rides inside without its FCS. The outer UDP source port is a hash of the inner headers, which gives the underlay ECMP entropy.
Go deeper
Layout
+----------------+------------+---------+----------+---------------------------------+
| Outer Ethernet | Outer IPv4 | UDP | VXLAN | Inner Ethernet frame |
| 14 bytes | 20 bytes | 8 bytes | 8 bytes | dst MAC, src MAC, type, payload |
+----------------+------------+---------+----------+---------------------------------+The 8-byte VXLAN header
| Flags (8 bits, I flag set) | Reserved (24 bits) |
| VNI (24 bits) | Reserved (8) |The I flag set to 1 means the VNI field is valid. Everything else is reserved and zero.
Outer header details
- Outer IP source is the local VTEP loopback. Destination is the remote VTEP, or a multicast group for BUM traffic in multicast mode.
- UDP destination 4789 is the IANA port. The Linux kernel defaults to 8472 (the pre-standard port)
unless you create the interface with
dstport 4789. - UDP source port comes from a hash of the inner frame. RFC 7348 recommends the 49152 to 65535 range.
- No fragmentation. VTEPs must not fragment VXLAN packets, so the underlay MTU has to fit them.
- The inner 802.1Q tag is normally removed: the VLAN is mapped to the VNI and recreated at the far end.
Overhead variations
| Underlay | Overhead |
|---|---|
| IPv4, untagged links | 50 bytes |
| IPv4, 802.1Q tagged links | 54 bytes |
| IPv6, untagged links | 70 bytes |
They will ask next
What happens to QoS markings?
Most VTEPs copy the inner DSCP into the outer IP header, so the underlay can queue VXLAN traffic by the tenant's marking. Check your platform's default, because the underlay only sees the outer header.
How is GENEVE different?
GENEVE uses UDP 6081, has an 8-byte base header plus variable-length TLV options and a protocol type field. The options let systems carry metadata such as policy group, which VXLAN cannot do without extensions.
Traps
- Saying the VXLAN header is 50 bytes. The VXLAN header is 8 bytes; 50 is the total overhead.
- Forgetting the extra 4 bytes when underlay links carry an 802.1Q tag.