# Resilience Engineering

Resilience is the ability of a system to adapt and recover in the face of disturbances, unexpected conditions, or failures. For complex systems, we can't prevent all failures - we must design to absorb, adapt, and recover.

## Resilience vs Reliability

| Aspect | Reliability | Resilience |
|---|---|---|
| **Focus** | "Does it work as specified?" | "Can it continue to function under stress/failure?" |
| **View of failure** | Failure is deviation from normal, should be prevented | Failure is normal in complex systems, must be managed |
| **Approach** | Prevention, redundancy, robustness | Adaptation, graceful degradation, recovery |
| **Question** | "How often does it fail?" | "How well does it handle failure?" |
| **Example** | "99.99% uptime" | "When DB is slow, we degrade gracefully and recover" |

## The Four Abilities of Resilience (Hollnagel)

| Ability | Description | Application |
|---|---|---|
| **Respond** | Know what to do when disruptions occur | Incident response, runbooks, on-call, fallbacks |
| **Monitor** | Know what to look for (early warning signs) | Observability, SLOs, anomaly detection, alerts |
| **Anticipate** | Know what to expect (potential disruptions) | Threat modeling, capacity planning, chaos engineering |
| **Learn** | Know what has happened and improve | Post-mortems, blameless culture, continuous improvement |

## Resilience Patterns

| Pattern | Purpose | Example |
|---|---|---|
| **Graceful Degradation** | Reduce functionality rather than fail completely | Return cached results, simpler response |
| **Circuit Breaker** | Prevent cascading failures | Stop calling failing service/model |
| **Bulkhead** | Isolate failures to one component | Separate thread pools, rate limits per tenant |
| **Timeout** | Don't wait forever | Set timeouts on all external calls |
| **Retry with Backoff** | Handle transient failures | Exponential backoff with jitter |
| **Fallback** | Alternative when primary fails | Cheaper model, cached response, human escalation |
| **Load Shedding** | Drop non-critical traffic under pressure | Reject low-priority requests when overloaded |
| **Backpressure** | Slow down producers when consumers overwhelmed | Queue depth limits, rate limiting |
| **Health Checks** | Detect unhealthy instances | Liveness/readiness probes |
| **Redundancy** | Multiple paths/components | Multi-region, replica models |

## Resilience for AI Systems

AI systems have unique resilience challenges:

| Challenge | Resilience Strategy |
|---|---|
| **Model unavailable** | Fallback to smaller model or cached response |
| **High latency / slow inference** | Timeout + fallback, streaming with timeout |
| **Rate limits / quota exhaustion** | Circuit breaker, queue with backpressure, model routing |
| **Hallucinations / bad outputs** | Guardrails, output validation, human-in-loop for high-stakes |
| **Cost spikes** | Budget limits, model routing to cheaper options, sampling |
| **Prompt injection** | Input guardrails, sandboxing, principle of least privilege |
| **Context window overflow** | Truncation strategies, summarization, chunking |
| **Tool failures** | Tool-specific timeouts/retries, graceful degradation |

## Chaos Engineering Principles

"Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production." - Principles of Chaos Engineering

| Principle | Application |
|---|---|
| **Start with hypothesis** | "If model X is unavailable, we should fall back to model Y within 5s" |
| **Vary real-world events** | Latency injection, errors, timeouts, network issues |
| **Run in production** | Safely, with blast radius control |
| **Automate experiments** | Run continuously |
| **Minimize blast radius** | Start small, have kill switch |
| **Learn and improve** | Fix issues found, add to runbooks |

### Simple Chaos Experiments for AI Systems
- **Latency injection**: Add delay to model API calls, verify timeouts/fallbacks work
- **Error injection**: Return 500s from model, verify circuit breaker opens
- **Tool failure**: Make a tool fail, verify agent handles gracefully
- **Rate limiting**: Simulate quota exceeded, verify backoff/routing
- **Guardrail trigger**: Send malicious input, verify blocking works

## Safety vs Resilience

| Concept | Focus |
|---|---|
| **Safety** | Prevent harm (injury, data breach, harmful content) |
| **Resilience** | Maintain function under stress/failure |
| **Security** | Protect against adversaries |
| **Reliability** | Consistent correct operation |

These overlap. A resilient system should also be safe and secure.

## Building a Resilient AI Platform

| Layer | Resilience Measures |
|---|---|
| **Model Serving** | Multi-model fallback, circuit breakers, autoscaling, health checks |
| **Retrieval/RAG** | Fallback to cached results, hybrid search, graceful degradation |
| **Agent/Orchestration** | Tool timeouts, max retries, human escalation, step limits |
| **Guardrails** | Input/output validation, PII detection, rate limiting |
| **Observability** | Tracing, metrics, alerts, evals on live traffic |
| **Operations** | Runbooks, on-call rotation, blameless post-mortems, chaos testing |

## Blameless Post-Mortems

Critical for learning in complex systems:

1. **Timeline**: What happened, when?
2. **Impact**: Users affected, duration, severity
3. **Root causes**: Look for systemic factors (not just "human error")
4. **What went well**: What helped us recover?
5. **What can be improved**: Action items with owners
6. **Lessons learned**: Share broadly

*Focus on "how did our system allow this?" not "who caused this?"*

## Related Notes

- [[Risk-Reliability/Risk-Analysis-FMEA-FTA.md]] — Proactive risk identification
- [[../AgenticEngineering/Production/Reliability.md]] — Reliability in production
- [[../Systems Engineering/Reliability/Circuit-Breakers-And-Fallbacks.md]] — Implementation patterns
- [[../Systems Engineering/Observability/]] — Monitoring for resilience
- [[../SoftEngine/Patterns/Resilience-Patterns.md]] — Software resilience patterns

## Sources

* Hollnagel, E. et al. - *Resilience Engineering in Practice*  
* Woods, D.D. & Hollnagel, E. - *Resilience Engineering: Concepts and Precepts*  
* Nygard, M.T. - *Release It! Design and Deploy Production-Ready Software* (2nd ed.)  
* Principles of Chaos Engineering (principlesofchaos.org)  
* Google SRE Book  
* AWS Well-Architected Framework - Reliability Pillar  
* Netflix Chaos Monkey / Simian Army