Starphix
Network Engineering / Sources
The Lab / Sources / Network Engineering / Pedagogy/Where-It-Goes-Wrong.md

Where It Goes Wrong

A taxonomy of the mistakes that actually happen in network engineering, sorted by where they originate. Every one of these is teachable in under two minutes and worth more than a definition.

The organising claim: networks rarely fail because a protocol is broken. They fail because a human made a reasonable decision with an incomplete model.

1. Conceptual mistakes (the model is wrong)

Mistake Why it happens Consequence The correction
"IP addresses identify machines" Every tutorial says so Multi-homed hosts, VIPs and containers confuse them Addresses identify interfaces
"The network delivers my message" Applications treat TCP as reliable end-to-end Assumes delivery, ordering and timing at the app layer TCP delivers a stream, eventually, or fails
"Inside the network is safe" Perimeter thinking Flat networks, ransomware spread Location is not identity
"Bandwidth is speed" Marketing Buys a bigger pipe, latency unchanged RTT and loss govern interactive speed
"Wi-Fi is wireless Ethernet" Same IP, same look Capacity planning by coverage Airtime is the shared resource
"DNS is a lookup" It looks like one Caching, TTL and anycast surprises DNS is a distributed, cached system
"Redundant means resilient" Diagrams show two links Both links in one conduit Redundancy needs independence and a test
"NAT is a firewall" Home routers conflate them Assumes protection that does not exist NAT is translation; policy is separate

Where these are born: the first hour of learning, and never revisited. Courses that skip the correction phase leave learners with a model that fails silently for years.

2. Address-planning mistakes (the design is wrong)

Mistake Where it happens Symptom Fix
Overlapping RFC 1918 across sites/VPN During M&A, or a first VPN Half the destinations unreachable, no error Renumber before the tunnel exists
VPC CIDR chosen from a default Cloud account creation Cannot peer with on-prem ever Plan the whole estate first
Same subnet on two VLANs Copy-paste config Intermittent reachability One subnet, one VLAN, always
Gateway outside the subnet Typo or a /25 in a /24 world Host cannot ARP the gateway Check mask on the host, not the router
Summarising wider than reality Route aggregation Blackhole for the gap Summarise only covering prefixes
No room to grow Tight allocation Renumbering project in year two Allocate by site, leave gaps

Where these are born: the design phase, usually on a whiteboard, usually under time pressure. They are the most expensive class because the fix touches every rule, ACL and DNS record made afterwards.

3. Configuration mistakes (the intent is right, the syntax is not)

Mistake Typical instance Detection
ACL order A permit above a deny Rule hit counters
Native VLAN mismatch Trunk between two vendors tcpdump -e, switch logs
Trunk allow-list missing a VLAN New VLAN "has no internet" show interfaces trunk, then ip route
BPDU Guard absent on access ports Unmanaged switch plugged in Broadcast storm, MAC flapping
OSPF reference bandwidth on one router Cost asymmetry show ip ospf interface, then compare costs
Static route shadowing dynamic Leftover from a migration show ip route, admin distance
NAT rule to a DHCP address Server moved Port forward points at nothing
Firewall rule for the wrong zone Traffic between two internal zones Packet capture on both sides
MTU left at 1,500 under a tunnel VXLAN, IPsec, PPPoE Small pings pass, large transfers hang
permit ip any any at the end of an ACL "To make it work" Policy review, hit counters

Where these are born: change windows, under time pressure, without a validation step. The structural fix is automation with pre-change tests, not more care.

4. Operational mistakes (the system is fine, the process is not)

Mistake What it looks like Prevention
Undocumented change "It worked yesterday" Change log, config archive, diff
No baseline Cannot tell attack from busy Metrics with history
Alert fatigue Real alert ignored Fewer, symptom-based alerts
No rollback plan Change made permanent by accident Canary plus a defined revert
Monitoring from inside only Customer sees what you cannot External synthetic checks
Certificate expiry Annual outage Expiry as a monitored metric
Clock drift Auth and TLS failures NTP as a dependency, monitored
Untested backup Restore fails when needed Restore drills
One person knows Bus factor of one Runbooks, rotation

Where these are born: in the gap between "we built it" and "we operate it". Most incidents are operational, which is why the last module of a course should be diagnosis, not protocols.

5. Security mistakes (the boundary is assumed)

Mistake Mechanism Real-world consequence
Flat internal network Any host reaches any host Ransomware encrypts the estate
Internet-exposed management SSH/RDP/database on a public IP Compromise within hours
Default credentials Devices shipped with admin/admin Botnet enrolment
Trusting X-Forwarded-For from anyone Header spoofing IP allowlists bypassed
No egress filtering Outbound is unrestricted Exfiltration and C2
Shared service credentials One key for many services One leak compromises all
Segmentation without policy VLANs, permit-any routing The diagram lies
RPKI not deployed Unverified BGP announcements Hijacks propagate

Where these are born: at the boundary between "make it work" and "make it safe", usually resolved in favour of "make it work".

6. The meta-mistake

Debugging by changing things. Every one of the above becomes harder to find when several variables move at once. The discipline that prevents most of the rest:

  1. Write down the last known good state.
  2. Change one thing.
  3. Record the result.
  4. If it did not help, revert before trying the next thing.

How to teach this

  • Two minutes per mistake, attached to the concept it belongs to. Do not collect them into a "common mistakes" appendix nobody reads.
  • Ask the learner to predict the symptom, not the fix. Prediction is where the model is tested.
  • Use one running scenario (an office, a website, a database) and let each mistake happen to it in turn.
  • Finish with the detection question: "what would have told you sooner?" That is the difference between an engineer and an operator.

From the Network Engineering research vault · CC BY 4.0 · all notes · raw markdown