# Observability

The ability to understand system state from its outputs. Three pillars + modern extensions.

---

## Three Pillars

| Pillar | Purpose | Key Characteristics |
|---|---|---|
| **Logs** | Discrete events with context | Structured (JSON); high cardinality; expensive at scale |
| **Metrics** | Aggregated measurements over time | Low cardinality; cheap; dashboards/alerts |
| **Traces** | Request flow across services | Distributed; latency analysis; dependency graph |

**Unified view**: Logs + Metrics + Traces = **Observability** (not monitoring)

---

## Logs

### Structured Logging (Required)
```json
{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "INFO",
  "service": "order-service",
  "trace_id": "abc123",
  "span_id": "def456",
  "message": "Order created",
  "order_id": "order-789",
  "customer_id": "cust-123",
  "amount_usd": 99.99
}
```

### Log Levels
| Level | Use For |
|---|---|
| **DEBUG** | Detailed diagnostic (dev only) |
| **INFO** | Business events, state changes |
| **WARN** | Unexpected but handled; potential issues |
| **ERROR** | Failure in current operation |
| **FATAL** | Process/service cannot continue |

### Best Practices
- **Structured (JSON)** — Parseable; queryable
- **Correlation IDs** — `trace_id`, `span_id` on every log line
- **Consistent fields** — `service`, `environment`, `version` always present
- **No secrets** — Never log passwords, tokens, PII, keys
- **Context-rich** — Include IDs, user context, decision rationale
- **Sample at high volume** — Tail-based sampling for DEBUG/TRACE

---

## Metrics

### Metric Types
| Type | Description | Example |
|---|---|---|
| **Counter** | Monotonically increasing | `http_requests_total` |
| **Gauge** | Point-in-time value | `memory_usage_bytes` |
| **Histogram** | Distribution of values | `http_request_duration_seconds` |
| **Summary** | Quantiles over sliding window | `request_latency_seconds` |

### Golden Signals (Google SRE)
| Signal | Metric | Alert Threshold |
|---|---|---|
| **Latency** | p50, p95, p99 request duration | p99 > SLO |
| **Traffic** | Requests/second (RPS) | Sudden drop/spike |
| **Errors** | Error rate (5xx / total) | > 1% or SLO breach |
| **Saturation** | Resource utilization % | > 80% CPU/mem/disk |

### RED Method (Per Service)
- **Rate** — Requests per second
- **Errors** — Failed requests per second
- **Duration** — Latency distribution (p50, p95, p99)

### USE Method (Per Resource)
- **Utilization** — % busy
- **Saturation** — Queue depth / wait time
- **Errors** — Error count

### Cardinality Management
- **Low cardinality** (metrics): `service`, `endpoint`, `status_code`, `region`
- **High cardinality** (logs/traces): `user_id`, `request_id`, `trace_id`
- **Never** high cardinality in metrics — explodes storage/cost

---

## Distributed Tracing

### Core Concepts
- **Trace** — End-to-end request flow (tree of spans)
- **Span** — Single operation (service call, DB query, cache lookup)
- **Context Propagation** — `traceparent` header (W3C TraceContext)

### Span Attributes (Minimum)
```json
{
  "span.name": "POST /orders",
  "span.kind": "SERVER",
  "service.name": "order-service",
  "http.method": "POST",
  "http.route": "/orders",
  "http.status_code": 201,
  "db.system": "postgresql",
  "db.statement": "INSERT INTO orders...",
  "error": false
}
```

### Sampling Strategies
| Strategy | Description | Use Case |
|---|---|---|
| **Head-based** | Decide at trace start | Simple; consistent |
| **Tail-based** | Decide after trace completes | Keep errors; drop success |
| **Adaptive** | Adjust rate based on throughput | High-volume services |

---

## Alerting

### Alert Design Principles
1. **Actionable** — Every alert requires human action
2. **Specific** — Clear what's wrong + where to look
3. **Routable** — Goes to right team
4. **Deduplicated** — One alert per incident

### Alert Categories
| Category | Response Time | Example |
|---|---|---|
| **Page (Critical)** | Immediate (24/7) | Service down; SLO breach; data loss |
| **Ticket (Warning)** | Business hours | Disk > 80%; latency elevated; queue backlog |
| **Log (Info)** | No response | New deployment; config change |

### Anti-Patterns
- **Alert fatigue** — Too many alerts → all ignored
- **Symptom alerts** — "CPU high" vs "Request latency p99 > 500ms"
- **No runbook** — Alert without documented response
- **Flaky alerts** — Intermittent; erodes trust

---

## Dashboards

### Dashboard Types
| Type | Audience | Content |
|---|---|---|
| **System Overview** | All engineers | Golden signals, deployment status |
| **Service Deep-Dive** | Service owners | RED metrics, dependencies, resources |
| **Business** | PMs, leadership | Conversion, revenue, active users |
| **Incident** | On-call | Current firing alerts, recent changes |

### Dashboard Principles
- **One purpose per dashboard**
- **Time-series aligned** — Same time range across panels
- **Threshold lines** — SLO targets visible
- **Drill-down links** — To logs, traces, related dashboards

---

## SLOs / SLIs / SLAs

| Term | Definition |
|---|---|
| **SLI** (Service Level Indicator) | Measurable metric (e.g., request latency < 200ms) |
| **SLO** (Service Level Objective) | Target for SLI (e.g., 99.9% of requests < 200ms) |
| **SLA** (Service Level Agreement) | Contract with consequences (external) |

### Error Budget
```
Error Budget = 1 - SLO
```
- 99.9% SLO → 0.1% error budget = 43.2 min/month downtime
- **Spend budget on releases** — If exhausted, freeze deployments
- **Alert on burn rate** — Fast burn = page; slow burn = ticket

---

## Implementation Stack

| Layer | Tools |
|---|---|
| **Collection** | OpenTelemetry (OTel) SDKs, Fluent Bit, Vector, Promtail |
| **Pipeline** | OpenTelemetry Collector, Kafka, Redis Streams |
| **Storage** | Loki (logs), Prometheus/Thanos/Cortex (metrics), Tempo/Jaeger (traces) |
| **Visualization** | Grafana, Kibana, Datadog, Honeycomb |
| **Alerting** | Alertmanager, Grafana Alerting, PagerDuty, Opsgenie |

### OpenTelemetry (Standard)
- **Vendor-neutral** — Single instrumentation, multiple backends
- **Auto-instrumentation** — Java, .NET, Node, Python, Go, etc.
- **Semantic Conventions** — Standard attribute names
- **Adopt early** — Instrument once, switch backends freely

---

## Cost Management

| Strategy | Description |
|---|---|
| **Tail-based sampling** | Keep 100% errors; sample success |
| **Metric aggregation** | Pre-aggregate in collector |
| **Retention policies** | Hot (7d) → Warm (30d) → Cold (1yr) |
| **Cardinality limits** | Drop high-cardinality labels |
| **Log level control** | DEBUG only in dev; INFO+ in prod |

---

## Related Notes

- [[DevOps/CI-CD-Best-Practices]] — Observability in pipeline
- [[Patterns/Resilience-Patterns]] — Circuit breaker, retry, timeout observability
- [[Performance/Performance-Engineering]] — Performance monitoring
- [[Security/Secure-Coding]] — Security logging
- [[Methodologies/Engineering-Culture]] — Blameless post-mortems need observability

---

*Sources: Google SRE Book, Observability Engineering (Charity Majors), OpenTelemetry, Prometheus, Grafana, Honeycomb, Datadog, CNCF Observability Whitepaper*