# NTP and Time

Time is a security dependency disguised as a convenience. Certificates, tokens, logs, replication and authentication all compare clocks, and all of them fail in ways that look like something else.

## Why clocks drift

Quartz oscillators drift by seconds per day, and virtual machines drift faster because they do not own the hardware clock. Without correction, a fleet disagrees by minutes within weeks.

## How NTP works

| Concept | Meaning |
|---|---|
| **Stratum** | Distance from a reference clock; 0 = atomic/GPS, 1 = directly attached, 2+ = downstream |
| **Offset** | Difference between local and reference time |
| **Delay** | Round-trip time to the server |
| **Slew** | Gradually adjust the clock rather than jumping |
| **Step** | Jump the clock (only when offset is large) |
| **NTS** | Network Time Security: authenticated NTP over TLS |

```bash
timedatectl                       # synchronized? which server? offset?
chronyc tracking                  # stratum, offset, drift, leap status
chronyc sources -v                # which servers, which are reachable
ntpq -p                           # classic NTP peers
```

UDP 123. Blocking it "because it is only time" is how a fleet ends up 40 seconds apart.

## What actually breaks with bad time

| Dependency | Tolerance | Symptom when broken |
|---|---|---|
| **TLS certificates** | Any skew | "certificate not yet valid" / "expired" on healthy certificates |
| **Kerberos / AD** | 5 minutes | Login failures, "clock skew too great" |
| **JWT / OAuth tokens** | Seconds | Tokens rejected as not-yet-valid or expired |
| **TOTP / MFA** | ~30 seconds | Codes "wrong" on one device |
| **Distributed databases** | Milliseconds | Replication lag, split-brain, inconsistent ordering |
| **Log correlation** | Seconds | The incident timeline is wrong, so the conclusion is wrong |
| **Caches / CDNs** | TTL semantics | Stale content served after an update |
| **Backups / cron** | Minutes | Jobs run twice or not at all across timezones/DST |

## Timezones, UTC, and DST

Store and transmit **UTC**. Convert only at the presentation layer. Systems that store local time produce two 01:30s on the DST changeover and an annual incident.

## Running NTP properly

1. Use at least four upstream sources (or your provider's endpoint, e.g. `169.254.169.123` on AWS, `metadata.google.internal` on GCP).
2. Run a local server for the fleet; do not point 500 hosts at `pool.ntp.org`.
3. Monitor `offset` and `stratum` as metrics; alert on `|offset| > 100 ms` and on `unsynchronised`.
4. Allow UDP 123 outbound to your time source only, and block it from user networks.
5. Enable NTS where supported; plain NTP can be spoofed.

## Where it goes wrong

- **NTP blocked by a firewall rule added for "security".** Nobody notices for months, then certificates fail during a routine deploy.
- **VMware/hypervisor time sync fighting NTP.** The guest steps the clock backwards and forward; services see time move backwards and corrupt state.
- **A single upstream server.** It goes down, everything drifts silently, and the first symptom is logins failing.
- **Stratum 16.** "Unsynchronised" is reported as stratum 16; a monitoring system that only checks "NTP is running" will not notice.
- **Clock jumps during operation.** A large step breaks timeouts, leases and token validity; slew instead.
- **Assuming timezone settings fix skew.** Timezone changes the display, not the offset from UTC.

## Teaching hook

**The office clock analogy.** Every clock in the building drifts, so once a day someone corrects them all from one accurate clock. If that one clock is wrong, or the corridor door is locked, every meeting in the building starts at a different wrong time — and nobody suspects the clock.

Demo: set a VM's clock five minutes back, then `curl` a HTTPS site and watch a valid certificate fail. Then `timedatectl set-ntp true` and watch it recover. The lesson sticks because the cause is invisible in the error message.

## Related Notes

- [[Core-Protocols/TLS-And-HTTPS.md]] — the most common victim
- [[Security/Network-Security-Fundamentals.md]] — time as an authentication input
- [[Automation-Observability/Telemetry-And-Monitoring.md]] — correlating logs
- [[Labs/Troubleshooting-Scenarios.md]] — the clock-skew scenario