Starphix
Network Engineering / Sources
The Lab / Sources / Network Engineering / Foundations/Packet-Flow-Encapsulation.md

Packet Flow and Encapsulation

One packet, followed from an application call to a wire and back. Everything else in networking is a detail of this journey, so it is the note to reread when a problem makes no sense.

The five-minute version

A program asks the operating system to send bytes to api.example.com:443. The OS resolves the name (DNS), picks a source port, wraps the bytes in a TCP segment, wraps that in an IP packet, wraps that in an Ethernet frame, and hands the frame to a network card that turns it into signals. Each router on the way strips the frame, looks at the destination IP, chooses a next hop, and builds a new frame. The destination reverses the process.

The one rule that prevents most confusion: IP addresses never change end to end; MAC addresses change at every hop.

Step by step

Step What happens Failure symptom if it breaks
1. Name resolution Stub resolver → recursive resolver → authoritative NS "Could not resolve host"
2. Route lookup Longest-prefix match in the routing table "Network unreachable"
3. Neighbour resolution ARP for the gateway (IPv4) or NDP (IPv6) "No route to host" on a link that is up
4. TCP handshake SYN → SYN-ACK → ACK Connection times out, or is refused
5. TLS handshake ClientHello, certificate, key exchange Certificate or protocol errors
6. Application exchange HTTP request, response, streaming body Slow, partial or 5xx responses
7. Teardown FIN/ACK each way, or RST Ports stuck in TIME_WAIT

What the headers actually say

Ethernet: [dst MAC][src MAC][type 0x0800][         payload        ][FCS]
IP:       [ver/ihl][tos][len][id][flags/frag][ttl][proto][csum][src IP][dst IP]
TCP:      [src port][dst port][seq][ack][flags][window][csum][options]
HTTP:     GET /v1/orders HTTP/1.1

Read bottom-up and the sizes add up: 14 bytes of Ethernet header, 20 of IPv4 (minimum), 20 of TCP (minimum), leaving 1,460 bytes of payload inside a 1,500-byte MTU. That 1,460 is the MSS, and it is where a whole family of "works on my machine, hangs on the VPN" bugs lives.

A real walkthrough

dig +short api.example.com            # 203.0.113.10
ip route get 203.0.113.10             # via 192.168.1.1 dev wlan0
arp -n 192.168.1.1                    # 192.168.1.1 -> aa:bb:cc:dd:ee:ff
ss -tin dst 203.0.113.10              # rtt, cwnd, retrans, mss
curl -v --max-time 5 https://api.example.com/health

Each command answers exactly one question. That is the discipline: do not debug the network, debug the step. If name resolution works and the route is right and ARP resolves and the handshake completes, the problem is above the transport layer and belongs to the application.

Encapsulation overhead, concretely

Configuration Bytes of overhead Consequence
Ethernet + IPv4 + TCP 54 Nothing to think about
802.1Q VLAN tag +4 1,496 max payload
VXLAN tunnel +50 Underlay MTU must be ~1,550+
IPsec ESP tunnel mode +50 to +73 Same MTU problem, plus fragmentation risk
PPPoE +8 1,492 usable, a classic residential edge surprise

Overhead is why tunnels break large file uploads while ping works: small packets fit, large ones do not, and if ICMP "fragmentation needed" is filtered, nothing tells you why.

Where it goes wrong

  • Debugging the wrong layer. People restart the app when ARP is stale, or blame DNS when the route is a blackhole. Follow the order; do not jump.
  • Assuming one name equals one IP. CDNs, load balancers and anycast return different answers by design. "It resolved to a different IP" is usually correct behaviour, not a bug.
  • Forgetting the reverse path. A packet that reaches the server is half a conversation. Asymmetric routing is invisible to ping and fatal to stateful firewalls.
  • Trusting ping as a health check. ICMP is often filtered and never exercises the port, TLS or application. Use a check that speaks the same protocol as the user.
  • Ignoring MTU until production. Test with a payload larger than 1,400 bytes before you ship anything that goes through a tunnel.

From the Network Engineering research vault · CC BY 4.0 · all notes · raw markdown