# Reliability

Guardrails, failover, safety, and incident response for AI systems in production.

## Core Concepts

| Concept | Description |
|---|---|
| **Guardrails** | Input/output filters that block unsafe or off-topic content |
| **Circuit Breaker** | Temporarily stop calling a failing model or dependency |
| **Fallback Model** | A smaller/cheaper model that takes over when the primary fails |
| **Retry with Backoff** | Exponential backoff for transient inference failures |
| **Error Budget** | Allowed failures before triggering remediation |
| **Incident Response** | Playbooks for AI-specific incidents: bad outputs, data leaks, cost spikes |

## Decision Guidance

| Situation | Recommended Approach |
|---|---|
| User-facing chatbot | Input guardrails (PII, jailbreak) + output guardrails (toxicity, hallucination) |
| Safety-critical system | Fallback model + human-in-the-loop for high-stakes outputs |
| High cost per request | Circuit breaker on expensive models; route to cheaper fallback |
| Model degradation | Error budget triggers rollback to previous model version |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| No guardrails on user input | Add PII and jailbreak detection before the model sees input |
| No fallback when model fails | Route to a smaller model or cached response on failure |
| Retrying forever on bad outputs | Circuit breaker; stop calling a consistently failing model |
| Deploying without an incident playbook | Write AI-specific runbooks: bad output, data leak, cost spike |

## Related Notes

- [[AgenticEngineering/Production/Reliability.md]] — Retries, fallbacks, circuit breakers, SLOs
- [[AgenticEngineering/Production/Security.md]] — Auth, authorization, data privacy, PII
- [[Observability/]] — Monitoring and alerting for reliability
- [[MLOps/]] — Rollback and deployment safety

## Sources

* Google SRE, NVIDIA NeMo Guardrails, LlamaGuard, Meta Purple Llama, AWS Bedrock Guardrails, Microsoft Guidance, AI incident response playbooks.*