Observability Stack

A complete observability architecture for AI systems: metrics, tracing, logging, alerting, and continuous evaluation across model, retrieval, and tool-calling layers.

Observability Pillars for AI

Pillar	Purpose	AI-Specific Signals	Tools
Metrics	Quantitative measurements over time	TTFT, TPOT, cost per token, error rate, cache hit rate	Prometheus, Datadog, CloudWatch
Traces	End-to-end request flow	Prompt → retrieval → generation → tool call → response	Tempo, Jaeger, OpenTelemetry, LangSmith
Logs	Structured events	Prompts, responses, retrieved chunks, tool outputs, guardrail decisions	Loki, ELK, CloudWatch, Datadog
Evaluation	Continuous quality measurement	LLM-as-judge scores, faithfulness, relevance, regression	Arize, LangSmith, Weights & Biases
Alerting	Proactive notification	Latency SLO breach, cost spike, guardrail violation, drift	Alertmanager, PagerDuty, Opsgenie

AI Trace Structure

graph TD
    A[User Request] --> B[API Gateway]
    B --> C[Input Guardrail]
    C --> D[Retrieval / RAG]
    D --> E[Prompt Construction]
    E --> F[Model Inference]
    F --> G[Tool Calling]
    G --> H[Output Guardrail]
    H --> I[Response to User]

    D --> D1[Vector DB Query]
    E --> E1[Prompt Tokens]
    F --> F1[TTFT / TPOT]
    F --> F2[Completion Tokens]
    G --> G1[Tool Latency]

    style A fill:#e1f5fe
    style I fill:#e8f5e9
    style C fill:#fff3e0
    style H fill:#fff3e0

Each span carries attributes: trace_id, span_id, model_name, prompt_tokens, completion_tokens, latency_ms, guardrail_verdict, retrieval_chunks, tool_name.

Decision Guidance

Situation	Recommended Approach
User-facing chatbot	Trace prompt→retrieval→generation→response; alert on p99 TTFT
RAG pipeline	Evaluate retrieval relevance and answer faithfulness continuously
Batch inference job	Track cost per token, throughput, error rate; alert on budget burn
Safety-critical system	Log guardrail violations and alert in real time

Anti-Patterns

Anti-Pattern	Fix
Only logging errors	Log prompts, responses, and retrieval context for debugging
No evaluation on live traffic	Add LLM-as-judge or regression tests on sampled production requests
Alerting on every spike	Use SLO-based alerting with error budgets; alert on sustained breaches
Correlating traces manually	Propagate trace ID through model, retrieval, and tool calls
No token-level cost visibility	Instrument prompt + completion tokens; report cost per request

Eval In Production — Continuous evaluation, LLM-as-judge, regression sets
Observability — Tracing, logging, metrics, alerting
Reliability — Guardrails, failover, incident response
MLOps — Deployment with observability built in
Knowledge Graphs — KG context for tracing

Structured Logging Example

import structlog
import time
from opentelemetry import trace

tracer = trace.get_tracer("ai-system")
logger = structlog.get_logger()

@tracer.start_as_current_span("generate_response")
async def generate_response(request):
    start = time.time()
    trace_id = format(trace.get_current_span().get_span_context().trace_id, "032x")

    logger.bind(
        trace_id=trace_id,
        user_id=request.user_id,
        model=request.model,
    )

    # Retrieval span
    with tracer.start_as_current_span("retrieval"):
        chunks = await retrieve(request.query)
        logger.info("retrieval_completed", chunk_count=len(chunks))

    # Inference span
    with tracer.start_as_current_span("inference"):
        response = await llm.generate(prompt)
        ttft_ms = (time.time() - start) * 1000
        logger.info(
            "inference_completed",
            prompt_tokens=response.prompt_tokens,
            completion_tokens=response.completion_tokens,
            ttft_ms=ttft_ms,
            cost_usd=response.cost_usd,
        )

    logger.bind(trace_id=trace_id).info("response_returned", duration_ms=ttft_ms)
    return response

Sources

OpenTelemetry, Honeycomb, LangSmith, Arize, Weights & Biases, Google SRE, AWS X-Ray, Datadog, LLM-as-judge research, OpenLLMetry.*

From the Systems Engineering research vault · CC BY 4.0 · all notes · raw markdown