Complexity, emergence, and AI systems
Some systems can be fully understood by taking them apart. Others can't — and the difference decides whether you should analyze or experiment.
- You can tell a complicated problem from a complex one, and pick the right response.
- You can apply systems discipline to AI features: canaries, eval gates, guardrails, fallbacks.
A jet engine is complicated: many parts, deep expertise required, but cause and effect are knowable and it can be fully decomposed. A support organization running an AI assistant is complex: users adapt, prompts drift, models change under you, incentives reshape behavior. You cannot analyze your way to the answer, because the answer changes while you're analyzing.
| Complicated | Complex | |
|---|---|---|
| Predictability | Understandable if you know the parts | Only explicable in hindsight |
| Decomposition | Works: reassemble and it behaves | Loses the property you care about |
| Failures | Known modes, analysable | Novel, emergent, from interactions |
| Approach | Analyse, design, verify | Probe safely, sense, respond, adapt |
| Example | Compiler, watch, payment switch | Traffic, markets, agent fleets, platform culture |
Your support assistant's answers are "fine" in the eval set and disliked in production. There is no single cause — users phrase things differently, agents trust it unevenly, the knowledge base drifts.
Response: probe. Ship a small variant to 5% of traffic, measure the outcome that matters (resolution without escalation), keep what works. Do not attempt a root-cause analysis of a moving target.
p95 latency doubled after a release. The system is knowable: queue depth, model call duration, cache hit rate.
Response: analyze. Get the trace, form a hypothesis, measure, fix, verify. Expertise and instrumentation solve this; experiments would be slower and noisier.
A guardrail is leaking personal data into responses, right now.
Response: act. Disable the path, revoke the credential, stop the bleeding. Then, once stable, move into analysis and make it a case for the risk register.
Rotating an API key, deploying the standard stack, running the migration you've run ten times.
Response: follow the procedure. Categorise, apply the known steps, automate it so nobody has to think about it again.
9.1What this changes for AI systems
An AI feature is usually a complex system with a complicated core: the model serving is analysable, the behavior in front of users is not. That combination argues for a specific set of practices, most of which are ordinary systems engineering applied to an unfamiliar component.
| Practice | What it is | Why systems thinking asks for it |
|---|---|---|
| Canary + eval gate | Route a small slice of traffic (5% is a common start) to the new model; promote only if an eval gate and SLOs hold | You can't verify a model's behavior on paper; you verify against live traffic |
| Continuous eval | Sample live traffic (1–5%) through a judge or regression set; alert on sustained score drops | Complex systems drift; a one-time gate decays |
| Guardrails in and out | Input checks (PII, injection), output checks (toxicity, groundedness, format) | The interface to users is the highest-consequence boundary |
| Fallback model | Second provider or smaller model behind the same interface | Reliability: a dependency you don't control will fail |
| Cost showback | Attribute token spend to team, feature and customer | Cost is a system property; unmeasured, it becomes an outage of a different kind |
| Trace per task | Every model call, tool call, decision and duration, with the exact context on failures | Emergent failures are only explicable with the interaction record |
None of these are AI-specific inventions. They are canary releases, acceptance sampling, input validation, redundancy, chargeback and tracing — the same patterns, applied where the component is probabilistic.
It looks rigorous: write a bigger eval set, find the root cause, fix it, declare victory. But in a complex domain the root cause moves. The tell is a team that keeps "fixing" the same class of problem every quarter. If the same failure returns in a new costume, stop analyzing and start changing the structure — incentives, interfaces, or the boundary itself.
A model update improves every offline metric, but users complain about tone. What is the right next move?
60-second recap
- Complicated: analyze it. Complex: probe it. Chaotic: stabilise first. Clear: follow the procedure.
- Emergence means interactions, not parts, produce the behavior you're fighting.
- AI systems are complex at the edges, so canary, eval continuously, gate with guardrails, keep a fallback.
- If the same failure keeps returning, change structure — not the prompt.