Risk, reliability, resilience
You can't prevent every failure. You can decide which ones you're willing to have, and what the system does in the ninety seconds after.
- You can score a failure mode with severity, occurrence and detection, and know when the score is lying to you.
- You can tell reliability apart from resilience, and name the pattern that addresses each.
Risk is not a feeling. It is a claim about two things: how bad it would be, and how likely it is. Written that way, risks become comparable, and mitigation becomes a budget decision instead of an argument.
6.1FMEA: score the failure, not the component
Failure mode and effects analysis walks each way a system can fail and scores three things from 1 to 10:
The product, S × O × D, is the risk priority number (RPN), range 1–1000. It is a rough sort, not a verdict — which is exactly why the calculator below shows the advice text, not just the number.
Try severity 10 — the advice should override the arithmetic. A catastrophic failure is not "acceptable because RPN is 60".
6.2Mitigate in this order
- Eliminate — remove the failure mode by design. Don't store the secret at all; don't allow the destructive action.
- Reduce — lower likelihood or severity: validation, limits, bulkheads, smaller blast radius.
- Detect — catch it earlier: health checks, canaries, output validation, alerting on the right signal.
- Recover — make failure survivable: retries with backoff, fallbacks, checkpoints, graceful degradation.
- Accept or transfer — consciously keep the risk, or move it (insurance, contract terms, vendor SLA). Write down that you chose this.
Most teams jump to step 4 because it's the one that fits in a sprint. But a retry on a failure you could have designed out is just a more expensive version of the same bug.
6.3Reliability is not resilience
| Reliability | Resilience | |
|---|---|---|
| Question | How often does it fail? | How well does it behave when it does? |
| View of failure | Deviation to be prevented | Normal, to be absorbed |
| Typical tools | Redundancy, testing, quality, capacity margin | Circuit breakers, fallbacks, degradation, runbooks |
| Measure | Uptime, MTBF, error rate | Time to detect, time to recover, blast radius |
Resilience engineering names four abilities worth building separately: respond (runbooks, on-call, fallbacks), monitor (know what to look for), anticipate (threat models, capacity planning, chaos experiments), learn (blameless post-mortems that change the system). A team with great monitoring and no authority to act has one of the four.
| Pattern | Failure it absorbs |
|---|---|
| Timeout | A dependency that never answers |
| Retry with backoff + jitter | Transient errors; also prevents synchronised retry storms |
| Circuit breaker | A dependency that is failing consistently — stop paying for it |
| Fallback | Primary gone: cheaper model, cached answer, human queue |
| Bulkhead | One tenant or feature consuming all capacity |
| Load shedding | Overload: drop low-priority work deliberately instead of randomly |
| Graceful degradation | Partial capability beats a blank error page |
| Checkpoint | Long work lost to a crash |
The dangerous dependencies are the ones nobody calls a dependency: the one DNS provider, the shared CI runner, the person who knows the deploy, the single API key, the spreadsheet that is actually the source of truth. Ask "what would take this down tomorrow?" and keep asking until the answers are boring.
An FMEA scores a failure mode S=8, O=6, D=9. Which change reduces risk the most for the least effort?
60-second recap
- Risk = severity × likelihood, written down with evidence so it can be compared.
- FMEA scores S, O, D. RPN sorts attention; severity 9–10 overrides the arithmetic.
- Mitigate in order: eliminate, reduce, detect, recover, accept.
- Reliability prevents failure; resilience survives it. You need both, and detection is usually the cheapest win.