# Observability Stack

A complete observability architecture for AI systems: metrics, tracing, logging, alerting, and continuous evaluation across model, retrieval, and tool-calling layers.

## Observability Pillars for AI

| Pillar | Purpose | AI-Specific Signals | Tools |
|---|---|---|---|
| **Metrics** | Quantitative measurements over time | TTFT, TPOT, cost per token, error rate, cache hit rate | Prometheus, Datadog, CloudWatch |
| **Traces** | End-to-end request flow | Prompt → retrieval → generation → tool call → response | Tempo, Jaeger, OpenTelemetry, LangSmith |
| **Logs** | Structured events | Prompts, responses, retrieved chunks, tool outputs, guardrail decisions | Loki, ELK, CloudWatch, Datadog |
| **Evaluation** | Continuous quality measurement | LLM-as-judge scores, faithfulness, relevance, regression | Arize, LangSmith, Weights & Biases |
| **Alerting** | Proactive notification | Latency SLO breach, cost spike, guardrail violation, drift | Alertmanager, PagerDuty, Opsgenie |

## AI Trace Structure

```mermaid
graph TD
    A[User Request] --> B[API Gateway]
    B --> C[Input Guardrail]
    C --> D[Retrieval / RAG]
    D --> E[Prompt Construction]
    E --> F[Model Inference]
    F --> G[Tool Calling]
    G --> H[Output Guardrail]
    H --> I[Response to User]
    
    D --> D1[Vector DB Query]
    E --> E1[Prompt Tokens]
    F --> F1[TTFT / TPOT]
    F --> F2[Completion Tokens]
    G --> G1[Tool Latency]
    
    style A fill:#e1f5fe
    style I fill:#e8f5e9
    style C fill:#fff3e0
    style H fill:#fff3e0
```

Each span carries attributes: `trace_id`, `span_id`, `model_name`, `prompt_tokens`, `completion_tokens`, `latency_ms`, `guardrail_verdict`, `retrieval_chunks`, `tool_name`.

## Decision Guidance

| Situation | Recommended Approach |
|---|---|
| User-facing chatbot | Trace prompt→retrieval→generation→response; alert on p99 TTFT |
| RAG pipeline | Evaluate retrieval relevance and answer faithfulness continuously |
| Batch inference job | Track cost per token, throughput, error rate; alert on budget burn |
| Safety-critical system | Log guardrail violations and alert in real time |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Only logging errors | Log prompts, responses, and retrieval context for debugging |
| No evaluation on live traffic | Add LLM-as-judge or regression tests on sampled production requests |
| Alerting on every spike | Use SLO-based alerting with error budgets; alert on sustained breaches |
| Correlating traces manually | Propagate trace ID through model, retrieval, and tool calls |
| No token-level cost visibility | Instrument prompt + completion tokens; report cost per request |

## Related Notes

- [[Observability/Eval-In-Production.md]] — Continuous evaluation, LLM-as-judge, regression sets
- [[AgenticEngineering/Production/Observability.md]] — Tracing, logging, metrics, alerting
- [[Reliability/]] — Guardrails, failover, incident response
- [[MLOps/]] — Deployment with observability built in
- [[AgenticEngineering/ContextEngineering/Knowledge-Graphs.md]] — KG context for tracing

## Structured Logging Example

```python
import structlog
import time
from opentelemetry import trace

tracer = trace.get_tracer("ai-system")
logger = structlog.get_logger()

@tracer.start_as_current_span("generate_response")
async def generate_response(request):
    start = time.time()
    trace_id = format(trace.get_current_span().get_span_context().trace_id, "032x")
    
    logger.bind(
        trace_id=trace_id,
        user_id=request.user_id,
        model=request.model,
    )
    
    # Retrieval span
    with tracer.start_as_current_span("retrieval"):
        chunks = await retrieve(request.query)
        logger.info("retrieval_completed", chunk_count=len(chunks))
    
    # Inference span
    with tracer.start_as_current_span("inference"):
        response = await llm.generate(prompt)
        ttft_ms = (time.time() - start) * 1000
        logger.info(
            "inference_completed",
            prompt_tokens=response.prompt_tokens,
            completion_tokens=response.completion_tokens,
            ttft_ms=ttft_ms,
            cost_usd=response.cost_usd,
        )
    
    logger.bind(trace_id=trace_id).info("response_returned", duration_ms=ttft_ms)
    return response
```

## Sources

* OpenTelemetry, Honeycomb, LangSmith, Arize, Weights & Biases, Google SRE, AWS X-Ray, Datadog, LLM-as-judge research, OpenLLMetry.*