When it breaks
Every professional network has failures. The difference between an engineer and everyone else is not knowing more protocols — it is having an order to work in.
- You can turn "the internet is broken" into a specific claim and test it in under a minute.
- You can name the five commands that separate name, path, port, trust and application failures.
A ticket arrives: the site is down. It contains no information. In the next sixty seconds you can either start guessing, or you can eliminate half the possible causes with three commands and know which half is left.
10.1The order that works
| Step | Question | Done when |
|---|---|---|
| 1. Define | Who, what, when, how often, since when? | You have a sentence a stranger could test |
| 2. Locate the layer | Name, path, port, trust, or application? | One layer is implicated and the ones below it are cleared |
| 3. Locate the path | Where does it stop? Same host, same subnet, same site, across the internet? | You know the last place it worked |
| 4. Read the evidence | What do the counters, logs and captures say — not the ticket? | You can state a cause, not a guess |
| 5. Change one thing | Does it fix it? Does it break something else? | The symptom changes, or it does not, and you know which |
| 6. Add the check | What would have caught this sooner? | An alert, a test, or a config guard exists |
Step 6 is the one people skip, and it is the difference between fixing an incident and preventing the next one. An outage that produced no new check will happen again, to somebody who has never seen it.
10.2Five commands, five questions
dig +short example.com # 1 name → an address, or nothing
ip route get 203.0.113.10 # 2 path → which interface and next hop
ping -c3 203.0.113.10 # 3 reach → does anything come back
nc -vz 203.0.113.10 443 # 4 port → open, refused, or silent
curl -vI https://example.com # 5 trust → certificate, TLS, HTTP statusRead them as a sequence, not a menu. Each one either clears a layer or implicates it, and the first one that fails is the one to work on. If dig returns nothing, the path does not matter. If the port is refused, the certificate does not matter.
Symptom: the API works from a laptop on the office network and fails from home over the VPN. It worked from home last week.
First command: ping -M do -s 1400 <api-host> from home.
What it proves: small pings succeed and the large one fails, so the path cannot carry a full-size packet. The VPN's MTU is smaller than the application assumes. Fix: lower the tunnel MTU or clamp MSS — not the application.
Symptom: a new service is unreachable from the application tier. The security group permits 8080 inbound. Nothing in the logs.
First command: nc -vz app.internal 8080 from the caller, then the same from another host in the same subnet.
What it proves: if it times out from one subnet and is refused from another, the difference is policy, not the service. Check the subnet-level rule set — a stateless ACL needs the ephemeral return path permitted, and the security group looks correct while it fails.
Symptom: everyone in one building is offline. The switch has link lights, the router reports the interface up, and the provider says nothing is wrong.
First command: ip neigh on the gateway, then check the DHCP scope.
What it proves: if clients hold 169.254. addresses, DHCP is not answering and the pool may be exhausted. The network is up; the service that hands out addresses is not. Existing leases keep working until they expire, which is why the failure spreads over hours rather than at once.
10.3The mistakes that make incidents longer
Two changes and the problem disappears: you have no idea which one fixed it, no idea what to keep, and no protection against it returning. Worse, you now believe the second change was necessary. Change one thing, record the result, and revert before trying the next — even when the pressure is to move fast, especially then.
"The network is slow" is a report from someone who cannot see the network. "The page takes nine seconds after login" is testable. The first step of every incident is converting the first sentence into the second, and it is usually a two-question conversation.
10.4What to do with this
You now have a model that covers the path from a cable to a certificate, and an order to work in when it fails. What you do not yet have is mileage, and mileage is the only thing that turns this into instinct.
Four weeks, deliberately small:
| Week | Do | You end up with |
|---|---|---|
| 1 | Run dig, ip route get, nc -vz and curl -v against five sites you use daily. Write down what each output told you. | Five annotated command transcripts |
| 2 | Build two Linux namespaces joined by a veth pair, ping across, then break it with a wrong mask and fix it. | A 20-line script that rebuilds the lab |
| 3 | Capture one page load with tcpdump and find the DNS query, the TCP handshake and the TLS ClientHello in the capture. | A capture file with three bookmarks |
| 4 | Take one real failure you have seen — yours or a public post-mortem — and write the six steps as they should have gone. | A one-page incident write-up |
Two habits matter more than the list. Read the evidence before changing anything — a capture, a counter, a log line, not a hunch. And write down what you found, because the network you understand today is the one you will be asked about in three months.
You will meet people who have configured switches for a decade and cannot explain why a packet takes the path it does. You will meet people who can explain it and have never configured anything. The useful engineer is the one who can do both, and the gap between those two groups is closed by exactly the four weeks above.
A connection attempt hangs and eventually times out. Which is true?
You have found and fixed the cause. What is the step people skip?
60-second recap
- Order beats knowledge. Define, locate the layer, locate the path, read evidence, change one thing.
- Refused is not timeout. One localises the fault to the service, the other to the path.
- Finish with a check. An incident that produced no new detection will happen again.