Overlays, SDN and VXLAN
Modern data-centre and cloud networks separate what the network appears to be from how packets actually move. That separation is the overlay/underlay split, and it is the idea behind VXLAN, EVPN, SDN controllers and service meshes.
Overlay vs underlay
| Underlay | Overlay | |
|---|---|---|
| What it is | The physical (or cloud-provider) IP fabric | Logical networks built on top |
| Addresses | Underlay IPs, one per VTEP | Tenant/segment IPs and MACs |
| Job | Move encapsulated packets | Provide isolation and mobility |
| Visibility | You can see it with traceroute |
Invisible without decapsulation |
The underlay only needs to deliver packets between tunnel endpoints. Everything the tenants care about — subnets, MACs, policy — lives in the overlay.
VXLAN in detail
VXLAN wraps a layer-2 frame in UDP so it can cross a layer-3 fabric.
[outer Ethernet][outer IP][UDP 4789][VXLAN 8B][inner Ethernet][inner IP][payload]
| Field | Size | Meaning |
|---|---|---|
| Flags | 1 byte | I flag set = VNI valid |
| Reserved | 3 bytes | |
| VNI | 3 bytes | 24-bit segment ID — 16.7 million networks |
| Reserved | 1 byte |
Header overhead is 50 bytes (14 outer Ethernet + 20 IP + 8 UDP + 8 VXLAN). That is why underlays are built with MTU 1,550–1,600, or with 9,000-byte jumbo frames.
VTEP (VXLAN Tunnel Endpoint) is the device that encapsulates and decapsulates. It can be a switch, a hypervisor, or a container network plugin.
EVPN: the control plane VXLAN needed
Classic VXLAN used flood-and-learn for MACs, which does not scale. EVPN distributes MAC and IP reachability via BGP, so the fabric knows where things are without flooding.
| Piece | Role |
|---|---|
| BGP EVPN | Control plane: advertises MAC/IP routes |
| VXLAN | Data plane: encapsulates frames |
| Anycast VTEP | Every leaf shares a loopback; traffic lands on the closest |
| Type-2 route | MAC/IP advertisement |
| Type-3 route | Inclusive multicast (BUM traffic) |
| Type-5 route | IP prefix route (for routing between subnets) |
This is the standard spine-leaf data-centre design, and what most cloud SDN fabrics look like underneath.
SDN in one page
Software-defined networking moves the control plane off the device and into a controller with a programmatic API.
| Model | Control plane | Examples |
|---|---|---|
| Traditional | Distributed, per device | IOS, Junos, FRR |
| Centralised SDN | Controller computes, devices forward | OpenDaylight, ONOS, NSX, ACI |
| Controller-less overlay | Distributed but API-managed | EVPN fabrics, Kubernetes CNI |
| Host-based | The hypervisor/OS does policy | OVN, Calico, Cilium |
The promise is intent-based policy and automation; the cost is a new distributed system to operate, and a control plane whose failure mode is "the network stops changing", not "the network stops working" — if it is designed correctly.
Kubernetes and container networking
Container networking is the overlay idea applied per pod:
- Every pod gets an IP; pods can reach each other without NAT (flat model).
- CNI plugins implement it: Calico (BGP or eBPF), Cilium (eBPF), Flannel (VXLAN), Weave.
- NetworkPolicy is the segmentation primitive; without it, every pod can reach every pod.
- Service IPs are virtual; kube-proxy or eBPF programs the data path.
The failure modes are the familiar ones: MTU too small for VXLAN + TLS, overlapping pod CIDRs across clusters, and no egress policy.
Where it goes wrong
- MTU not raised on the underlay. Ping works, TLS handshakes work, large uploads hang. VXLAN plus ESP plus PPPoE is how you lose 200 bytes and a weekend.
- Flood-and-learn at scale. MAC tables overflow, and every new host floods the fabric.
- Underlay routing errors are invisible. The overlay shows "tunnel down" while the actual problem is an IGP adjacency three hops away.
- Overlapping VNIs across tenants. Isolation is a configuration claim, not a cryptographic one.
- MTU mismatch between cloud and on-prem. Cloud gateways often cap lower than the data-centre fabric.
- Assuming the overlay is encrypted. VXLAN is encapsulation, not security; anyone on the underlay can read it.
- CNI pod CIDR overlap with the office network, making a VPN impossible without renumbering.
Related Notes
- VPC Fundamentals — the provider's version of this
- VPNs And Tunnels — encapsulation and MTU
- Network Design And Topologies — spine-leaf as a design
- NetDevOps Fundamentals — configuring fabric as code