# Observability

Monitoring, tracing, logging, alerting, and continuous evaluation for AI systems in production.

## Core Concepts

| Concept | Description |
|---|---|
| **Metrics** | Numerical signals over time: latency, throughput, error rate, cost per token |
| **Traces** | End-to-end request flows across model, retrieval, and tool calls |
| **Logs** | Structured events: prompts, responses, retrieval chunks, tool outputs |
| **Evaluation** | Continuous LLM-as-judge, regression sets, correctness checks on live traffic |
| **Drift Detection** | Detecting shifts in input distribution or model output quality |
| **SLOs** | Service-level objectives for latency, availability, correctness |

## Decision Guidance

| Situation | Recommended Approach |
|---|---|
| User-facing chatbot | Trace prompt→retrieval→generation→response; alert on p99 latency |
| RAG pipeline | Evaluate retrieval relevance and answer faithfulness continuously |
| Batch inference job | Track cost per token, throughput, error rate; alert on budget burn |
| Safety-critical system | Guardrail violations logged and alerted in real time |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Only logging errors | Log prompts, responses, and retrieval context for debugging |
| No eval on live traffic | Add LLM-as-judge or regression tests on sampled production requests |
| Alerting on every spike | Use SLO-based alerting with error budgets |
| Correlating traces manually | Use a trace ID propagated through model, retrieval, and tool calls |

## Related Notes

- [[AgenticEngineering/Production/Observability.md]] — Tracing, logging, metrics, debugging
- [[Reliability/]] — Guardrails, failover, incident response
- [[MLOps/]] — Deployment with observability built in
- [[AgenticEngineering/ContextEngineering/Knowledge-Graphs.md]] — KG context for tracing

## Sources

* OpenTelemetry, Honeycomb, LangSmith, Arize, Weights & Biases, Google SRE, AWS X-Ray, Datadog, LLM-as-judge research.*