# Overlays, SDN and VXLAN

Modern data-centre and cloud networks separate **what the network appears to be** from **how packets actually move**. That separation is the overlay/underlay split, and it is the idea behind VXLAN, EVPN, SDN controllers and service meshes.

## Overlay vs underlay

| | Underlay | Overlay |
|---|---|---|
| What it is | The physical (or cloud-provider) IP fabric | Logical networks built on top |
| Addresses | Underlay IPs, one per VTEP | Tenant/segment IPs and MACs |
| Job | Move encapsulated packets | Provide isolation and mobility |
| Visibility | You can see it with `traceroute` | Invisible without decapsulation |

The underlay only needs to deliver packets between tunnel endpoints. Everything the tenants care about — subnets, MACs, policy — lives in the overlay.

## VXLAN in detail

VXLAN wraps a layer-2 frame in UDP so it can cross a layer-3 fabric.

```text
[outer Ethernet][outer IP][UDP 4789][VXLAN 8B][inner Ethernet][inner IP][payload]
```

| Field | Size | Meaning |
|---|---|---|
| Flags | 1 byte | I flag set = VNI valid |
| Reserved | 3 bytes | |
| **VNI** | 3 bytes | 24-bit segment ID — 16.7 million networks |
| Reserved | 1 byte | |

Header overhead is **50 bytes** (14 outer Ethernet + 20 IP + 8 UDP + 8 VXLAN). That is why underlays are built with MTU 1,550–1,600, or with 9,000-byte jumbo frames.

**VTEP** (VXLAN Tunnel Endpoint) is the device that encapsulates and decapsulates. It can be a switch, a hypervisor, or a container network plugin.

## EVPN: the control plane VXLAN needed

Classic VXLAN used flood-and-learn for MACs, which does not scale. **EVPN** distributes MAC and IP reachability via BGP, so the fabric knows where things are without flooding.

| Piece | Role |
|---|---|
| **BGP EVPN** | Control plane: advertises MAC/IP routes |
| **VXLAN** | Data plane: encapsulates frames |
| **Anycast VTEP** | Every leaf shares a loopback; traffic lands on the closest |
| **Type-2 route** | MAC/IP advertisement |
| **Type-3 route** | Inclusive multicast (BUM traffic) |
| **Type-5 route** | IP prefix route (for routing between subnets) |

This is the standard spine-leaf data-centre design, and what most cloud SDN fabrics look like underneath.

## SDN in one page

Software-defined networking moves the **control plane** off the device and into a controller with a programmatic API.

| Model | Control plane | Examples |
|---|---|---|
| Traditional | Distributed, per device | IOS, Junos, FRR |
| Centralised SDN | Controller computes, devices forward | OpenDaylight, ONOS, NSX, ACI |
| Controller-less overlay | Distributed but API-managed | EVPN fabrics, Kubernetes CNI |
| Host-based | The hypervisor/OS does policy | OVN, Calico, Cilium |

The promise is intent-based policy and automation; the cost is a new distributed system to operate, and a control plane whose failure mode is "the network stops changing", not "the network stops working" — if it is designed correctly.

## Kubernetes and container networking

Container networking is the overlay idea applied per pod:

- Every pod gets an IP; pods can reach each other without NAT (**flat model**).
- **CNI** plugins implement it: Calico (BGP or eBPF), Cilium (eBPF), Flannel (VXLAN), Weave.
- **NetworkPolicy** is the segmentation primitive; without it, every pod can reach every pod.
- **Service** IPs are virtual; kube-proxy or eBPF programs the data path.

The failure modes are the familiar ones: MTU too small for VXLAN + TLS, overlapping pod CIDRs across clusters, and no egress policy.

## Where it goes wrong

- **MTU not raised on the underlay.** Ping works, TLS handshakes work, large uploads hang. VXLAN plus ESP plus PPPoE is how you lose 200 bytes and a weekend.
- **Flood-and-learn at scale.** MAC tables overflow, and every new host floods the fabric.
- **Underlay routing errors are invisible.** The overlay shows "tunnel down" while the actual problem is an IGP adjacency three hops away.
- **Overlapping VNIs across tenants.** Isolation is a configuration claim, not a cryptographic one.
- **MTU mismatch between cloud and on-prem.** Cloud gateways often cap lower than the data-centre fabric.
- **Assuming the overlay is encrypted.** VXLAN is encapsulation, not security; anyone on the underlay can read it.
- **CNI pod CIDR overlap** with the office network, making a VPN impossible without renumbering.

## Teaching hook

**The shipping-container analogy.** The underlay is the road network; the overlay is the shipping container. The road only needs to carry containers between ports (VTEPs); what is inside, and whose it is, is a matter for the container manifest (VNI and EVPN routes).

Demo: two Linux hosts, `ip link add vxlan0 type vxlan id 42 ...`, put an address on each, and ping across a routed network. Then lower the underlay MTU and watch a large transfer fail.

## Related Notes

- [[Cloud-Modern-Networking/VPC-Fundamentals.md]] — the provider's version of this
- [[Security/VPNs-And-Tunnels.md]] — encapsulation and MTU
- [[Foundations/Network-Design-And-Topologies.md]] — spine-leaf as a design
- [[Automation-Observability/NetDevOps-Fundamentals.md]] — configuring fabric as code