The explosive proliferation of containerized workloads, distributed Kubernetes clusters, and east-west microservice communications has rendered traditional three-tier hierarchical data center networks (Core, Distribution, Access) completely obsolete. Classic Layer 2 Spanning Tree Protocol (STP) architectures block up to 50% of available bandwidth to prevent switching loops, suffer from agonizingly slow convergence during topology changes, and collapse under large-scale virtual machine mobility. Modern hyperscale and enterprise data centers deploy non-blocking Clos Leaf-Spine topologies decoupled into an underlying high-speed IP routing layer (Underlay) and an agile virtualized network layer (Overlay) driven by BGP-EVPN and VXLAN (RFC 7348). In this technical monograph, we dissect the mathematical foundations of Clos fabrics, VXLAN encapsulation, BGP-EVPN route types (RFC 7432), and Distributed Anycast Default Gateways.
Data Center Fabric Engineering Reference
Engineered by the Enterprise Network & Cloud Infrastructure Editorial Board. Architecture baselines: RFC 7938 (Use of BGP for Routing in Large-Scale Data Centers), RFC 7348 (Virtual eXtensible Local Area Network), RFC 7432 (BGP MPLS-Based Ethernet VPN), and RFC 8365 (BGP EVPN for VXLAN). Target: Zero packet drop across ECMP hash paths with deterministic 1-hop leaf-to-spine east-west latency.
1. Clos Leaf-Spine Topologies: Eliminating Spanning Tree Bottlenecks
Modern data center fabrics are built upon the multi-stage switching network concepts pioneered by Charles Clos in 1953. In a two-tier Clos fabric, every leaf switch (Top-of-Rack / ToR) connects to every spine switch in a full mesh. Leaf switches do not connect to other leaves, and spine switches do not connect to other spines. This deterministic physical geometry guarantees that any two servers attached to different leaf switches are separated by exactly three physical hops (Leaf → Spine → Leaf), ensuring predictable latency and uniform bisectional bandwidth across the entire compute farm.
Unlike STP topologies that place redundant uplinks into an inactive blocking state, leaf switches leverage Equal-Cost Multi-Pathing (ECMP) to hash outbound flows across all available spine switches concurrently. If an enterprise deploys four 400G spine switches, a 32-port 100G leaf switch with four 100G uplinks utilizes 100% of its 400 Gbps aggregate uplink bandwidth across the active spine plane.
2. Underlay Routing Mechanics: Single-AS vs. eBGP-per-Leaf Patterns
The physical underlay network's sole responsibility is providing high-speed, loop-free point-to-point IP transport between leaf and spine loopback addresses (VTEPs). While interior link-state protocols like OSPF or IS-IS can serve as the underlay IGP, hyperscale data center designs predominantly adopt eBGP in the underlay (RFC 7938) due to superior scale, predictable fault isolation, and simplified prefix suppression:
- eBGP-per-Rack (Recommended): Each leaf pair shares a unique private 2-byte or 4-byte ASN, while all spine switches share a single common ASN. BGP AS-Path loop detection naturally prevents routing loops, while standard ECMP balances flows across spines without requiring complicated IGP tuning.
- Unnumbered BGP: Eliminates the requirement to allocate and document thousands of point-to-point /30 or /31 subnets across leaf-spine links by running BGP directly over IPv6 link-local addresses (RFC 5549), using extended next-hop encoding to advertise IPv4 prefixes over IPv6 peering sessions.
3. VXLAN Encapsulation (RFC 7348) & The Virtual Tunnel Endpoint
VXLAN provides Layer 2 overlay network virtualization over Layer 3 IP underlays. VXLAN encapsulates an entire original Ethernet frame inside a standard UDP packet (destination UDP port 4789). The key components of VXLAN include:
- VXLAN Network Identifier (VNI): A 24-bit segment identifier replacing the legacy 12-bit IEEE 802.1Q VLAN tag. VNIs expand maximum segment scale from 4,094 VLANs to over 16.7 million isolated virtual networks.
- Virtual Tunnel Endpoint (VTEP): The hardware entity (typically the leaf switch ASIC) responsible for encapsulating originating Ethernet frames into IP/UDP packets and decapsulating incoming packets before delivering them to local server ports.
- UDP Entropy Hashing: The source UDP port in the outer header is dynamically calculated by hashing the inner Ethernet frame's Layer 2/3/4 headers. This allows underlay spine switches to load-balance VXLAN traffic across ECMP paths without inspecting the encapsulated payload.
4. BGP-EVPN Control Plane (RFC 7432): Eliminating Flood-and-Learn
Early VXLAN implementations relied on IP multicast in the underlay to handle flood-and-learn discovery of MAC addresses, creating severe operational overhead and multicast state bloat. Ethernet VPN (EVPN) coupled with MP-BGP provides a standardized, scalable control plane that advertises MAC and IP reachability directly through BGP routing updates:
| EVPN Route Type | Standard Name | RFC Reference | Operational Function in VXLAN Fabric |
|---|---|---|---|
| Type-1 | Ethernet Auto-Discovery | RFC 7432 | Discovers multi-homed leaf switches (ESI) and enables fast failover convergence. |
| Type-2 | MAC / IP Advertisement | RFC 7432 | Advertises end-host MAC and optional ARP IP address bound to a specific VTEP IP. |
| Type-3 | Inclusive Multicast Ethernet Tag | RFC 7432 | Sets up head-end replication tunnels for Broadcast, Unknown Unicast & Multicast (BUM). |
| Type-4 | Ethernet Segment Route | RFC 7432 | Elects Designated Forwarder (DF) on dual-homed multi-chassis leaf links (ESI-LAG). |
| Type-5 | IP Prefix Route | RFC 7432 / 9136 | Advertises inter-subnet or routed external prefixes without advertising individual host MACs. |
5. Distributed Anycast Default Gateways & Symmetric IRB
In legacy networks, first-hop default gateways were concentrated on centralized distribution switches using VRRP or HSRP, forcing all inter-VLAN traffic to hairpin across the core. In a modern EVPN-VXLAN fabric, every leaf switch acts as the default gateway for all subnets simultaneously through Distributed Anycast Gateway architecture.
Every leaf switch is configured with the identical virtual IP address and virtual MAC address across each VNI. When a virtual machine or container transmits an ARP request for its gateway, the local ToR leaf responds immediately at wire rate. When routing between different VNIs, the fabric executes Symmetric Integrated Routing and Bridging (IRB): the ingress leaf switch routes the packet from the source VNI to a tenant-specific Layer 3 VNI, encapsulates it to the destination leaf VTEP, and the egress leaf routes the packet from the L3 VNI into the target destination subnet, ensuring zero triangular routing.
6. Technical Comparison: Legacy 3-Tier STP vs. EVPN-VXLAN Leaf-Spine
| Architectural Vector | Legacy 3-Tier STP Network | BGP-EVPN VXLAN Leaf-Spine Fabric |
|---|---|---|
| Link Utilization | Active/Standby (50% bandwidth blocked by STP) | 100% Active/Active ECMP across all spine uplinks |
| Failure Domain | VLAN-wide broadcast storms affect entire DC | Isolated per VNI; broadcast contained via BGP control plane |
| Segmentation Scale | Max 4,094 VLANs (802.1Q 12-bit limit) | 16.7+ Million VNIs (VXLAN 24-bit identifier) |
| East-West Latency | Variable (2 to 6 hops depending on STP root) | Deterministic 3 hops (Leaf → Spine → Leaf) |
| Host Discovery | Data-plane broadcast flood-and-learn | Control-plane MP-BGP Type-2 Route updates |
| Default Gateway | Centralized HSRP/VRRP active-standby pairs | Distributed Anycast Gateway active on every ToR leaf |
7. Production Switch Configuration: Arista EOS EVPN-VXLAN Leaf
Below is a production Arista EOS configuration for a ToR Leaf switch establishing a VXLAN overlay with EVPN control plane and Distributed Anycast Gateway:
! ==============================================================================
! Arista EOS EVPN-VXLAN Leaf Switch Configuration
! ==============================================================================
service routing protocols model multi-agent
!
vlan 10
name Web_Tier
!
interface Vlan10
description Web_Tier_Distributed_Gateway
ip address virtual 10.1.10.1/24
!
interface Loopback0
description Underlay_Router_ID
ip address 10.255.1.11/32
!
interface Loopback1
description VTEP_Source_IP
ip address 10.254.1.11/32
!
interface Vxlan1
vxlan source-interface Loopback1
vxlan udp-port 4789
vxlan vlan 10 vni 10010
!
router bgp 65011
router-id 10.255.1.11
neighbor SPINE_PEERS peer group
neighbor SPINE_PEERS remote-as 65000
neighbor SPINE_PEERS send-community extended
neighbor 10.255.0.1 peer group SPINE_PEERS
neighbor 10.255.0.2 peer group SPINE_PEERS
!
address-family evpn
neighbor SPINE_PEERS activate
!
vlan 10
rd 65011:10010
route-target both 10010:10010
redistribute learned
!Frequently Asked Questions (FAQ)
Why is multi-agent routing model required for EVPN on Arista EOS?
Arista EOS traditionally ran separate discrete routing engines for standard IP protocols. BGP-EVPN requires continuous bidirectional coordination between the BGP routing daemon, the Layer 2 forwarding engine (MAC table), and the VXLAN VTEP hardware driver. Running service routing protocols model multi-agent enables the multi-protocol EVPN daemon (bgp-evpn) required to synchronize Layer 2 and Layer 3 forwarding state into the hardware forwarding information base.
What is the difference between Symmetric and Asymmetric IRB in VXLAN?
In Asymmetric IRB, the ingress leaf routes and bridges packets to the destination VNI before sending them across the VXLAN tunnel, requiring every leaf to maintain routing state and ARP entries for all VNIs in the entire data center. In Symmetric IRB, the ingress leaf routes packets into an intermediate tenant Layer 3 VNI (transit VNI), and the egress leaf routes the packet from the L3 VNI into the target local VNI. Symmetric IRB dramatically reduces MAC and ARP table memory pressure on leaf switches.
How does EVPN handle Broadcast, Unknown Unicast, and Multicast (BUM) traffic without multicast?
EVPN eliminates the requirement for PIM multicast in the underlay through Ingress Replication (Head-End Replication). When a leaf receives a BUM frame from a local server, it queries its EVPN Type-3 Inclusive Multicast route table. The leaf then creates individual point-to-point unicast copies of the frame and transmits them directly to every remote VTEP participating in that VNI, eliminating underlay multicast complexity.
What is an Ethernet Segment Identifier (ESI) in EVPN?
An Ethernet Segment Identifier (ESI) is a unique 10-byte identifier assigned to a server dual-homed across two separate leaf switches (Active-Active multi-homing). EVPN uses ESI routes (Type-1 and Type-4) to coordinate split-horizon loop prevention, perform designated forwarder election for BUM traffic, and allow both leaf switches to forward traffic concurrently without legacy proprietary protocols like Cisco vPC or Arista MLAG.