NTP and Time
Time is a security dependency disguised as a convenience. Certificates, tokens, logs, replication and authentication all compare clocks, and all of them fail in ways that look like something else.
Why clocks drift
Quartz oscillators drift by seconds per day, and virtual machines drift faster because they do not own the hardware clock. Without correction, a fleet disagrees by minutes within weeks.
How NTP works
| Concept | Meaning |
|---|---|
| Stratum | Distance from a reference clock; 0 = atomic/GPS, 1 = directly attached, 2+ = downstream |
| Offset | Difference between local and reference time |
| Delay | Round-trip time to the server |
| Slew | Gradually adjust the clock rather than jumping |
| Step | Jump the clock (only when offset is large) |
| NTS | Network Time Security: authenticated NTP over TLS |
timedatectl # synchronized? which server? offset?
chronyc tracking # stratum, offset, drift, leap status
chronyc sources -v # which servers, which are reachable
ntpq -p # classic NTP peers
UDP 123. Blocking it "because it is only time" is how a fleet ends up 40 seconds apart.
What actually breaks with bad time
| Dependency | Tolerance | Symptom when broken |
|---|---|---|
| TLS certificates | Any skew | "certificate not yet valid" / "expired" on healthy certificates |
| Kerberos / AD | 5 minutes | Login failures, "clock skew too great" |
| JWT / OAuth tokens | Seconds | Tokens rejected as not-yet-valid or expired |
| TOTP / MFA | ~30 seconds | Codes "wrong" on one device |
| Distributed databases | Milliseconds | Replication lag, split-brain, inconsistent ordering |
| Log correlation | Seconds | The incident timeline is wrong, so the conclusion is wrong |
| Caches / CDNs | TTL semantics | Stale content served after an update |
| Backups / cron | Minutes | Jobs run twice or not at all across timezones/DST |
Timezones, UTC, and DST
Store and transmit UTC. Convert only at the presentation layer. Systems that store local time produce two 01:30s on the DST changeover and an annual incident.
Running NTP properly
- Use at least four upstream sources (or your provider's endpoint, e.g.
169.254.169.123on AWS,metadata.google.internalon GCP). - Run a local server for the fleet; do not point 500 hosts at
pool.ntp.org. - Monitor
offsetandstratumas metrics; alert on|offset| > 100 msand onunsynchronised. - Allow UDP 123 outbound to your time source only, and block it from user networks.
- Enable NTS where supported; plain NTP can be spoofed.
Where it goes wrong
- NTP blocked by a firewall rule added for "security". Nobody notices for months, then certificates fail during a routine deploy.
- VMware/hypervisor time sync fighting NTP. The guest steps the clock backwards and forward; services see time move backwards and corrupt state.
- A single upstream server. It goes down, everything drifts silently, and the first symptom is logins failing.
- Stratum 16. "Unsynchronised" is reported as stratum 16; a monitoring system that only checks "NTP is running" will not notice.
- Clock jumps during operation. A large step breaks timeouts, leases and token validity; slew instead.
- Assuming timezone settings fix skew. Timezone changes the display, not the offset from UTC.
Related Notes
- TLS And HTTPS — the most common victim
- Network Security Fundamentals — time as an authentication input
- Telemetry And Monitoring — correlating logs
- Troubleshooting Scenarios — the clock-skew scenario