Observability
Monitoring, tracing, logging, alerting, and continuous evaluation for AI systems in production.
Core Concepts
| Concept |
Description |
| Metrics |
Numerical signals over time: latency, throughput, error rate, cost per token |
| Traces |
End-to-end request flows across model, retrieval, and tool calls |
| Logs |
Structured events: prompts, responses, retrieval chunks, tool outputs |
| Evaluation |
Continuous LLM-as-judge, regression sets, correctness checks on live traffic |
| Drift Detection |
Detecting shifts in input distribution or model output quality |
| SLOs |
Service-level objectives for latency, availability, correctness |
Decision Guidance
| Situation |
Recommended Approach |
| User-facing chatbot |
Trace prompt→retrieval→generation→response; alert on p99 latency |
| RAG pipeline |
Evaluate retrieval relevance and answer faithfulness continuously |
| Batch inference job |
Track cost per token, throughput, error rate; alert on budget burn |
| Safety-critical system |
Guardrail violations logged and alerted in real time |
Anti-Patterns
| Anti-Pattern |
Fix |
| Only logging errors |
Log prompts, responses, and retrieval context for debugging |
| No eval on live traffic |
Add LLM-as-judge or regression tests on sampled production requests |
| Alerting on every spike |
Use SLO-based alerting with error budgets |
| Correlating traces manually |
Use a trace ID propagated through model, retrieval, and tool calls |
- Observability — Tracing, logging, metrics, debugging
- Reliability — Guardrails, failover, incident response
- MLOps — Deployment with observability built in
- Knowledge Graphs — KG context for tracing
Sources
- OpenTelemetry, Honeycomb, LangSmith, Arize, Weights & Biases, Google SRE, AWS X-Ray, Datadog, LLM-as-judge research.*
