# Network Telemetry and Monitoring

You cannot operate what you cannot see, and most networks are monitored by a graph nobody trusts. This note is about what to measure, how to collect it, and the failure modes of monitoring itself.

## Push vs pull

| | Polling (SNMP, ICMP, API) | Streaming telemetry |
|---|---|---|
| Direction | Collector asks | Device pushes |
| Interval | 1–5 minutes | Seconds or sub-second |
| Granularity | Counters and gauges | Interfaces, queues, hardware, flows |
| Scale | Every device, cheaply | Needs a collector and a pipeline |
| Detection | After the interval | Near real time |

SNMP is not dead — it is the lowest common denominator and every device supports it. Streaming telemetry (gNMI/gRPC, OpenConfig, IPFIX/NetFlow) is what you use when the question is "what happened in the last 30 seconds".

## The four golden signals, applied to networks

| Signal | Network version | Where |
|---|---|---|
| **Latency** | RTT, application response time | Probes, synthetic transactions, flow data |
| **Traffic** | Bits/s, packets/s, connections | Interface counters, NetFlow |
| **Errors** | CRC, drops, retransmits, resets | Interface counters, TCP stats |
| **Saturation** | Utilisation, queue depth, table occupancy | Interfaces, `conntrack`, MAC/ARP/route tables |

Add the network-specific fifth: **reachability**. A path can be uncongested and wrong.

## Counters that actually diagnose

| Counter | Means | Common cause |
|---|---|---|
| `input errors` / CRC | Frames damaged | Cable, optics, duplex |
| `output drops` | Egress queue full | Oversubscription, microbursts |
| `late collisions` | Duplex mismatch | Half/full mismatch |
| `runt` / `giant` | Bad framing | MTU mismatch, bad NIC |
| `no buffer` | Buffer exhaustion | Burst, not average |
| `resets` (TCP) | Flows killed | Firewall, timeout, application |
| `retransmits` | Loss or reordering | Congestion, wireless, middlebox |

Averages hide microbursts. A 10 Gbps link at "40% average" can be 100% for 200 ms, and that is where the latency complaints come from. Look at p95/p99 and at queue drops, not at the daily graph.

## Logs, flows, and traces

| Data | Answers | Cost |
|---|---|---|
| **Syslog** | "What did the device say?" | Cheap, unstructured |
| **SNMP traps** | "Something changed" | Cheap, lossy (UDP) |
| **Flow (NetFlow/IPFIX/sFlow)** | "Who talked to whom, how much" | Medium, sampled |
| **Packet capture** | "What exactly was on the wire?" | Expensive, short window |
| **Synthetic probes** | "Does it work from here?" | Cheap, partial |
| **Distributed traces** | "Where did the request spend time?" | Needs instrumentation |

The hierarchy matters: metrics tell you **something** is wrong, logs tell you **what**, traces tell you **where**, packets tell you **why**. Escalate down that list rather than starting with capture.

## Alerting that people keep

- Alert on **symptoms users feel** (latency, error rate), not on every counter.
- Every alert needs a **runbook link** and an **owner**.
- Use **multi-window burn rates** for error budgets instead of fixed thresholds.
- Suppress alerts during known maintenance, and record why.
- Delete alerts nobody has acted on in six months; they train people to ignore the pager.

## Monitoring the monitoring

- Is the collector alive? Is the pipeline lagging?
- Are devices still sending? A silent device and a healthy device look the same in a dashboard.
- Is the clock synchronised? Correlation is meaningless without it.
- Is the retention long enough for the incident you will have?

## Where it goes wrong

- **Dashboards with 40 panels.** Nobody reads them; the one that mattered was at the bottom.
- **Alerting on CPU.** Network devices run hot by design; CPU 80% is often fine, drops are not.
- **Polling every 5 minutes and calling it real time.** An outage that lasted 90 seconds never appears.
- **Averages over microbursts.** Capacity planning on daily graphs oversubscribes the busy hours.
- **No baseline.** Without a "normal" week, every anomaly is ambiguous.
- **Logs collected, never parsed.** Retention without structure is a storage bill.
- **Monitoring from one vantage point.** A probe inside the network cannot see what a customer sees.

## Teaching hook

**The car dashboard analogy.** Speed and fuel are what you watch (golden signals); the engine warning light is an alert; the diagnostic port is telemetry; and the fact that the fuel gauge is broken is a monitoring problem of its own.

Exercise: give learners a week of interface counters for one link with a 40% average and a 100% microburst, and ask whether to upgrade. The average says no; the queue drops say yes.

## Related Notes

- [[Automation-Observability/NetDevOps-Fundamentals.md]] — deploying the collectors as code
- [[Automation-Observability/Packet-Analysis-Wireshark.md]] — the last resort
- [[Network-Services/NTP-And-Time.md]] — correlation depends on it
- [[Labs/Troubleshooting-Scenarios.md]] — applying the signals