Resilience Engineering
Resilience is the ability of a system to adapt and recover in the face of disturbances, unexpected conditions, or failures. For complex systems, we can't prevent all failures - we must design to absorb, adapt, and recover.
Resilience vs Reliability
| Aspect |
Reliability |
Resilience |
| Focus |
"Does it work as specified?" |
"Can it continue to function under stress/failure?" |
| View of failure |
Failure is deviation from normal, should be prevented |
Failure is normal in complex systems, must be managed |
| Approach |
Prevention, redundancy, robustness |
Adaptation, graceful degradation, recovery |
| Question |
"How often does it fail?" |
"How well does it handle failure?" |
| Example |
"99.99% uptime" |
"When DB is slow, we degrade gracefully and recover" |
The Four Abilities of Resilience (Hollnagel)
| Ability |
Description |
Application |
| Respond |
Know what to do when disruptions occur |
Incident response, runbooks, on-call, fallbacks |
| Monitor |
Know what to look for (early warning signs) |
Observability, SLOs, anomaly detection, alerts |
| Anticipate |
Know what to expect (potential disruptions) |
Threat modeling, capacity planning, chaos engineering |
| Learn |
Know what has happened and improve |
Post-mortems, blameless culture, continuous improvement |
Resilience Patterns
| Pattern |
Purpose |
Example |
| Graceful Degradation |
Reduce functionality rather than fail completely |
Return cached results, simpler response |
| Circuit Breaker |
Prevent cascading failures |
Stop calling failing service/model |
| Bulkhead |
Isolate failures to one component |
Separate thread pools, rate limits per tenant |
| Timeout |
Don't wait forever |
Set timeouts on all external calls |
| Retry with Backoff |
Handle transient failures |
Exponential backoff with jitter |
| Fallback |
Alternative when primary fails |
Cheaper model, cached response, human escalation |
| Load Shedding |
Drop non-critical traffic under pressure |
Reject low-priority requests when overloaded |
| Backpressure |
Slow down producers when consumers overwhelmed |
Queue depth limits, rate limiting |
| Health Checks |
Detect unhealthy instances |
Liveness/readiness probes |
| Redundancy |
Multiple paths/components |
Multi-region, replica models |
Resilience for AI Systems
AI systems have unique resilience challenges:
| Challenge |
Resilience Strategy |
| Model unavailable |
Fallback to smaller model or cached response |
| High latency / slow inference |
Timeout + fallback, streaming with timeout |
| Rate limits / quota exhaustion |
Circuit breaker, queue with backpressure, model routing |
| Hallucinations / bad outputs |
Guardrails, output validation, human-in-loop for high-stakes |
| Cost spikes |
Budget limits, model routing to cheaper options, sampling |
| Prompt injection |
Input guardrails, sandboxing, principle of least privilege |
| Context window overflow |
Truncation strategies, summarization, chunking |
| Tool failures |
Tool-specific timeouts/retries, graceful degradation |
Chaos Engineering Principles
"Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production." - Principles of Chaos Engineering
| Principle |
Application |
| Start with hypothesis |
"If model X is unavailable, we should fall back to model Y within 5s" |
| Vary real-world events |
Latency injection, errors, timeouts, network issues |
| Run in production |
Safely, with blast radius control |
| Automate experiments |
Run continuously |
| Minimize blast radius |
Start small, have kill switch |
| Learn and improve |
Fix issues found, add to runbooks |
Simple Chaos Experiments for AI Systems
- Latency injection: Add delay to model API calls, verify timeouts/fallbacks work
- Error injection: Return 500s from model, verify circuit breaker opens
- Tool failure: Make a tool fail, verify agent handles gracefully
- Rate limiting: Simulate quota exceeded, verify backoff/routing
- Guardrail trigger: Send malicious input, verify blocking works
Safety vs Resilience
| Concept |
Focus |
| Safety |
Prevent harm (injury, data breach, harmful content) |
| Resilience |
Maintain function under stress/failure |
| Security |
Protect against adversaries |
| Reliability |
Consistent correct operation |
These overlap. A resilient system should also be safe and secure.
| Layer |
Resilience Measures |
| Model Serving |
Multi-model fallback, circuit breakers, autoscaling, health checks |
| Retrieval/RAG |
Fallback to cached results, hybrid search, graceful degradation |
| Agent/Orchestration |
Tool timeouts, max retries, human escalation, step limits |
| Guardrails |
Input/output validation, PII detection, rate limiting |
| Observability |
Tracing, metrics, alerts, evals on live traffic |
| Operations |
Runbooks, on-call rotation, blameless post-mortems, chaos testing |
Blameless Post-Mortems
Critical for learning in complex systems:
- Timeline: What happened, when?
- Impact: Users affected, duration, severity
- Root causes: Look for systemic factors (not just "human error")
- What went well: What helped us recover?
- What can be improved: Action items with owners
- Lessons learned: Share broadly
Focus on "how did our system allow this?" not "who caused this?"
- Risk Analysis FMEA FTA — Proactive risk identification
- Reliability — Reliability in production
- Circuit Breakers And Fallbacks — Implementation patterns
- Observability — Monitoring for resilience
- Resilience Patterns — Software resilience patterns
Sources
- Hollnagel, E. et al. - Resilience Engineering in Practice
- Woods, D.D. & Hollnagel, E. - Resilience Engineering: Concepts and Precepts
- Nygard, M.T. - Release It! Design and Deploy Production-Ready Software (2nd ed.)
- Principles of Chaos Engineering (principlesofchaos.org)
- Google SRE Book
- AWS Well-Architected Framework - Reliability Pillar
- Netflix Chaos Monkey / Simian Army
