# Troubleshooting Scenarios

Ten failures, written the way they arrive: a symptom, no context, and a user waiting. Each one names the first question, the evidence that settles it, and the lesson that generalises.

The method underneath all ten: **narrow the layer, then narrow the path, then read the evidence.** Never change two things at once.

## 1. "The site is down"

- **First question:** down for everyone, or one person?
- **Evidence:** `curl -v` from outside and inside; check DNS separately (`dig +short`); check the certificate dates.
- **Typical cause:** expired certificate (everyone), or a stale DNS record after a migration (some people, depending on resolver cache).
- **Lesson:** "down" is a claim, not a diagnosis. Split it into name, path, port, TLS, application — in that order.

## 2. "It works from home but not from the office"

- **First question:** what does the internal client resolve the hostname to?
- **Evidence:** `dig` inside vs outside; `curl -v` inside; check for hairpin NAT or split-horizon DNS.
- **Typical cause:** the name resolves to the public IP inside, and the gateway does not hairpin back in.
- **Lesson:** internal access is a DNS and NAT design question, not a firewall bug.

## 3. "The VPN connects but nothing loads"

- **First question:** does `ping` the gateway work, and does `ping` a large payload work?
- **Evidence:** `ping -M do -s 1400`, `ip route`, `traceroute`, MTU on the tunnel interface.
- **Typical cause:** MTU black hole — small packets pass, large ones vanish, TLS handshakes hang.
- **Lesson:** "connected" only means the tunnel exists. Test with realistic payload sizes.

## 4. "Intermittent packet loss, everyone blames the network"

- **First question:** one host, one subnet, or one application?
- **Evidence:** `mtr` for 200 packets to the destination; interface counters for CRC/drops; check duplex with `ethtool`.
- **Typical cause:** duplex mismatch or a failing cable on one link — invisible to `ping` at low rates.
- **Lesson:** loss that appears under load and not at idle is a physical or duplex problem, not routing.

## 5. "The server can't reach the database"

- **First question:** does the TCP handshake complete?
- **Evidence:** `nc -vz db 5432` (refused vs timeout), `ss -tn`, security group and NACL rules, host firewall.
- **Typical cause:** a cloud NACL that permits inbound on the port but not outbound on ephemeral ports.
- **Lesson:** refused and timeout mean different things. Timeout means something dropped it silently — find which layer.

## 6. "Logins fail on some machines, not others"

- **First question:** what is the clock offset on the failing machines?
- **Evidence:** `timedatectl`, `chronyc tracking`, compare with the domain controller.
- **Typical cause:** clock skew beyond five minutes breaking Kerberos, or expired client certificates.
- **Lesson:** authentication failures are often time failures. Check the clock before the directory.

## 7. "The new VLAN has no internet"

- **First question:** can the host ping its default gateway by IP?
- **Evidence:** `ip route`, ARP/neighbour table, switch trunk allowed-VLAN list, SVI existence, DHCP scope.
- **Typical cause:** the VLAN exists on the access switch but is not in the trunk's allowed list, or there is no SVI/relay.
- **Lesson:** layer 2 problems present as layer 3 symptoms. Verify the gateway before blaming routing.

## 8. "Traffic takes the slow path after a change"

- **First question:** which route is installed, and why that one?
- **Evidence:** `ip route get <dest>`, `show ip route`, admin distance, longest-prefix match, OSPF cost or BGP local preference.
- **Typical cause:** a leftover static route with AD 1 beating OSPF, or a reference-bandwidth mismatch making every link cost 1.
- **Lesson:** routing incidents are usually tie-breaks. Read the tie-break rules, not the topology.

## 9. "The network went down when someone plugged in a switch"

- **First question:** is there a broadcast storm (CPU, MAC flapping, all links saturated)?
- **Evidence:** interface utilisation, `show spanning-tree`, MAC flap logs, BPDU Guard err-disabled ports.
- **Typical cause:** an unmanaged switch on an access port creating a loop; no BPDU Guard.
- **Lesson:** loops are an access-layer problem with an access-layer fix. PortFast plus BPDU Guard, everywhere.

## 10. "Intermittent 502s behind the load balancer"

- **First question:** are the 502s correlated with one backend, or with a deploy?
- **Evidence:** backend health check history, per-backend latency and error rate, connection counts, draining settings.
- **Typical cause:** a health check that passes while the application is broken, or connections dropped during a deploy because draining is misconfigured.
- **Lesson:** the load balancer is a state machine with opinions about health. Read its health, not the app's logs alone.

## The general method

1. **Define the failure precisely** — who, what, when, how often, since when.
2. **Pick the layer** with the cheapest discriminating test.
3. **Change one variable** and record the result.
4. **Fix the cause, not the symptom** — then write down what would have detected it sooner.
5. **Add the check** that fails next time: an alert, a health check, a test, a config guard.

## Where it goes wrong

- **Changing several things at once.** The problem disappears and nobody knows why, so it returns.
- **Trusting the ticket's diagnosis.** "The network is slow" is a symptom report from someone who cannot see the network.
- **Skipping the cheap test.** `curl -v` and `dig` cost five seconds and eliminate half the possibilities.
- **Not writing it down.** The same incident recurs in six months with a different on-call engineer.
- **Fixing the alert instead of the cause.** Raising a threshold is not a repair.

## Teaching hook

Present each scenario with only the first line — the complaint — and ask for the next single command. The discipline of choosing one discriminating test is the skill; the fix is a detail.

## Related Notes

- [[Foundations/Packet-Flow-Encapsulation.md]] — the step list these scenarios walk
- [[Labs/Home-Lab-Setup.md]] — reproduce them safely
- [[Automation-Observability/Telemetry-And-Monitoring.md]] — detect them sooner
- [[Network-Services/NTP-And-Time.md]] — scenario 6 in detail