Starphix
Network Engineering / Sources
The Lab / Sources / Network Engineering / Automation-Observability/Telemetry-And-Monitoring.md

Network Telemetry and Monitoring

You cannot operate what you cannot see, and most networks are monitored by a graph nobody trusts. This note is about what to measure, how to collect it, and the failure modes of monitoring itself.

Push vs pull

Polling (SNMP, ICMP, API) Streaming telemetry
Direction Collector asks Device pushes
Interval 1–5 minutes Seconds or sub-second
Granularity Counters and gauges Interfaces, queues, hardware, flows
Scale Every device, cheaply Needs a collector and a pipeline
Detection After the interval Near real time

SNMP is not dead — it is the lowest common denominator and every device supports it. Streaming telemetry (gNMI/gRPC, OpenConfig, IPFIX/NetFlow) is what you use when the question is "what happened in the last 30 seconds".

The four golden signals, applied to networks

Signal Network version Where
Latency RTT, application response time Probes, synthetic transactions, flow data
Traffic Bits/s, packets/s, connections Interface counters, NetFlow
Errors CRC, drops, retransmits, resets Interface counters, TCP stats
Saturation Utilisation, queue depth, table occupancy Interfaces, conntrack, MAC/ARP/route tables

Add the network-specific fifth: reachability. A path can be uncongested and wrong.

Counters that actually diagnose

Counter Means Common cause
input errors / CRC Frames damaged Cable, optics, duplex
output drops Egress queue full Oversubscription, microbursts
late collisions Duplex mismatch Half/full mismatch
runt / giant Bad framing MTU mismatch, bad NIC
no buffer Buffer exhaustion Burst, not average
resets (TCP) Flows killed Firewall, timeout, application
retransmits Loss or reordering Congestion, wireless, middlebox

Averages hide microbursts. A 10 Gbps link at "40% average" can be 100% for 200 ms, and that is where the latency complaints come from. Look at p95/p99 and at queue drops, not at the daily graph.

Logs, flows, and traces

Data Answers Cost
Syslog "What did the device say?" Cheap, unstructured
SNMP traps "Something changed" Cheap, lossy (UDP)
Flow (NetFlow/IPFIX/sFlow) "Who talked to whom, how much" Medium, sampled
Packet capture "What exactly was on the wire?" Expensive, short window
Synthetic probes "Does it work from here?" Cheap, partial
Distributed traces "Where did the request spend time?" Needs instrumentation

The hierarchy matters: metrics tell you something is wrong, logs tell you what, traces tell you where, packets tell you why. Escalate down that list rather than starting with capture.

Alerting that people keep

  • Alert on symptoms users feel (latency, error rate), not on every counter.
  • Every alert needs a runbook link and an owner.
  • Use multi-window burn rates for error budgets instead of fixed thresholds.
  • Suppress alerts during known maintenance, and record why.
  • Delete alerts nobody has acted on in six months; they train people to ignore the pager.

Monitoring the monitoring

  • Is the collector alive? Is the pipeline lagging?
  • Are devices still sending? A silent device and a healthy device look the same in a dashboard.
  • Is the clock synchronised? Correlation is meaningless without it.
  • Is the retention long enough for the incident you will have?

Where it goes wrong

  • Dashboards with 40 panels. Nobody reads them; the one that mattered was at the bottom.
  • Alerting on CPU. Network devices run hot by design; CPU 80% is often fine, drops are not.
  • Polling every 5 minutes and calling it real time. An outage that lasted 90 seconds never appears.
  • Averages over microbursts. Capacity planning on daily graphs oversubscribes the busy hours.
  • No baseline. Without a "normal" week, every anomaly is ambiguous.
  • Logs collected, never parsed. Retention without structure is a storage bill.
  • Monitoring from one vantage point. A probe inside the network cannot see what a customer sees.

From the Network Engineering research vault · CC BY 4.0 · all notes · raw markdown