Observability

The ability to understand system state from its outputs. Three pillars + modern extensions.

Three Pillars

Unified view: Logs + Metrics + Traces = Observability (not monitoring)

Logs

Structured Logging (Required)

{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "INFO",
  "service": "order-service",
  "trace_id": "abc123",
  "span_id": "def456",
  "message": "Order created",
  "order_id": "order-789",
  "customer_id": "cust-123",
  "amount_usd": 99.99
}

Log Levels

Best Practices

• Structured (JSON) — Parseable; queryable
• Correlation IDs — trace_id, span_id on every log line
• Consistent fields — service, environment, version always present
• No secrets — Never log passwords, tokens, PII, keys
• Context-rich — Include IDs, user context, decision rationale
• Sample at high volume — Tail-based sampling for DEBUG/TRACE

Metrics

Metric Types

Golden Signals (Google SRE)

RED Method (Per Service)

• Rate — Requests per second
• Errors — Failed requests per second
• Duration — Latency distribution (p50, p95, p99)

USE Method (Per Resource)

• Utilization — % busy
• Saturation — Queue depth / wait time
• Errors — Error count

Cardinality Management

• Low cardinality (metrics): service, endpoint, status_code, region
• High cardinality (logs/traces): user_id, request_id, trace_id
• Never high cardinality in metrics — explodes storage/cost

Distributed Tracing

Core Concepts

• Trace — End-to-end request flow (tree of spans)
• Span — Single operation (service call, DB query, cache lookup)
• Context Propagation — traceparent header (W3C TraceContext)

Span Attributes (Minimum)

{
  "span.name": "POST /orders",
  "span.kind": "SERVER",
  "service.name": "order-service",
  "http.method": "POST",
  "http.route": "/orders",
  "http.status_code": 201,
  "db.system": "postgresql",
  "db.statement": "INSERT INTO orders...",
  "error": false
}

Sampling Strategies

Alerting

Alert Design Principles

1. Actionable — Every alert requires human action
2. Specific — Clear what's wrong + where to look
3. Routable — Goes to right team
4. Deduplicated — One alert per incident

Alert Categories

Anti-Patterns

• Alert fatigue — Too many alerts → all ignored
• Symptom alerts — "CPU high" vs "Request latency p99 > 500ms"
• No runbook — Alert without documented response
• Flaky alerts — Intermittent; erodes trust

Dashboards

Dashboard Types

Dashboard Principles

• One purpose per dashboard
• Time-series aligned — Same time range across panels
• Threshold lines — SLO targets visible
• Drill-down links — To logs, traces, related dashboards

SLOs / SLIs / SLAs

Error Budget

Error Budget = 1 - SLO

• 99.9% SLO → 0.1% error budget = 43.2 min/month downtime
• Spend budget on releases — If exhausted, freeze deployments
• Alert on burn rate — Fast burn = page; slow burn = ticket

Implementation Stack

OpenTelemetry (Standard)

• Vendor-neutral — Single instrumentation, multiple backends
• Auto-instrumentation — Java, .NET, Node, Python, Go, etc.
• Semantic Conventions — Standard attribute names
• Adopt early — Instrument once, switch backends freely

Cost Management

Related Notes

• CI CD Best Practices — Observability in pipeline
• Resilience Patterns — Circuit breaker, retry, timeout observability
• Performance Engineering — Performance monitoring
• Secure Coding — Security logging
• Engineering Culture — Blameless post-mortems need observability

Sources: Google SRE Book, Observability Engineering (Charity Majors), OpenTelemetry, Prometheus, Grafana, Honeycomb, Datadog, CNCF Observability Whitepaper