Starphix
Network Engineering / Sources
The Lab / Sources / Network Engineering / Labs/Troubleshooting-Scenarios.md

Troubleshooting Scenarios

Ten failures, written the way they arrive: a symptom, no context, and a user waiting. Each one names the first question, the evidence that settles it, and the lesson that generalises.

The method underneath all ten: narrow the layer, then narrow the path, then read the evidence. Never change two things at once.

1. "The site is down"

  • First question: down for everyone, or one person?
  • Evidence: curl -v from outside and inside; check DNS separately (dig +short); check the certificate dates.
  • Typical cause: expired certificate (everyone), or a stale DNS record after a migration (some people, depending on resolver cache).
  • Lesson: "down" is a claim, not a diagnosis. Split it into name, path, port, TLS, application — in that order.

2. "It works from home but not from the office"

  • First question: what does the internal client resolve the hostname to?
  • Evidence: dig inside vs outside; curl -v inside; check for hairpin NAT or split-horizon DNS.
  • Typical cause: the name resolves to the public IP inside, and the gateway does not hairpin back in.
  • Lesson: internal access is a DNS and NAT design question, not a firewall bug.

3. "The VPN connects but nothing loads"

  • First question: does ping the gateway work, and does ping a large payload work?
  • Evidence: ping -M do -s 1400, ip route, traceroute, MTU on the tunnel interface.
  • Typical cause: MTU black hole — small packets pass, large ones vanish, TLS handshakes hang.
  • Lesson: "connected" only means the tunnel exists. Test with realistic payload sizes.

4. "Intermittent packet loss, everyone blames the network"

  • First question: one host, one subnet, or one application?
  • Evidence: mtr for 200 packets to the destination; interface counters for CRC/drops; check duplex with ethtool.
  • Typical cause: duplex mismatch or a failing cable on one link — invisible to ping at low rates.
  • Lesson: loss that appears under load and not at idle is a physical or duplex problem, not routing.

5. "The server can't reach the database"

  • First question: does the TCP handshake complete?
  • Evidence: nc -vz db 5432 (refused vs timeout), ss -tn, security group and NACL rules, host firewall.
  • Typical cause: a cloud NACL that permits inbound on the port but not outbound on ephemeral ports.
  • Lesson: refused and timeout mean different things. Timeout means something dropped it silently — find which layer.

6. "Logins fail on some machines, not others"

  • First question: what is the clock offset on the failing machines?
  • Evidence: timedatectl, chronyc tracking, compare with the domain controller.
  • Typical cause: clock skew beyond five minutes breaking Kerberos, or expired client certificates.
  • Lesson: authentication failures are often time failures. Check the clock before the directory.

7. "The new VLAN has no internet"

  • First question: can the host ping its default gateway by IP?
  • Evidence: ip route, ARP/neighbour table, switch trunk allowed-VLAN list, SVI existence, DHCP scope.
  • Typical cause: the VLAN exists on the access switch but is not in the trunk's allowed list, or there is no SVI/relay.
  • Lesson: layer 2 problems present as layer 3 symptoms. Verify the gateway before blaming routing.

8. "Traffic takes the slow path after a change"

  • First question: which route is installed, and why that one?
  • Evidence: ip route get <dest>, show ip route, admin distance, longest-prefix match, OSPF cost or BGP local preference.
  • Typical cause: a leftover static route with AD 1 beating OSPF, or a reference-bandwidth mismatch making every link cost 1.
  • Lesson: routing incidents are usually tie-breaks. Read the tie-break rules, not the topology.

9. "The network went down when someone plugged in a switch"

  • First question: is there a broadcast storm (CPU, MAC flapping, all links saturated)?
  • Evidence: interface utilisation, show spanning-tree, MAC flap logs, BPDU Guard err-disabled ports.
  • Typical cause: an unmanaged switch on an access port creating a loop; no BPDU Guard.
  • Lesson: loops are an access-layer problem with an access-layer fix. PortFast plus BPDU Guard, everywhere.

10. "Intermittent 502s behind the load balancer"

  • First question: are the 502s correlated with one backend, or with a deploy?
  • Evidence: backend health check history, per-backend latency and error rate, connection counts, draining settings.
  • Typical cause: a health check that passes while the application is broken, or connections dropped during a deploy because draining is misconfigured.
  • Lesson: the load balancer is a state machine with opinions about health. Read its health, not the app's logs alone.

The general method

  1. Define the failure precisely — who, what, when, how often, since when.
  2. Pick the layer with the cheapest discriminating test.
  3. Change one variable and record the result.
  4. Fix the cause, not the symptom — then write down what would have detected it sooner.
  5. Add the check that fails next time: an alert, a health check, a test, a config guard.

Where it goes wrong

  • Changing several things at once. The problem disappears and nobody knows why, so it returns.
  • Trusting the ticket's diagnosis. "The network is slow" is a symptom report from someone who cannot see the network.
  • Skipping the cheap test. curl -v and dig cost five seconds and eliminate half the possibilities.
  • Not writing it down. The same incident recurs in six months with a different on-call engineer.
  • Fixing the alert instead of the cause. Raising a threshold is not a repair.

From the Network Engineering research vault · CC BY 4.0 · all notes · raw markdown