Production: money, safety, sleep
A working demo and a system you can leave running overnight are separated by four things: cost control, observability, failure handling, and permissions.
- You can estimate what an agent costs before you build it, and know what to log when it misbehaves.
- You know the seven reliability patterns that keep a long-running task from becoming an incident.
9.1Where the money goes
For most agent systems, tokens dominate the bill — not because any single call is expensive, but because every iteration resends the context and tasks run for many iterations.
Remember that an agent task can be 5–20 model calls, each resending a context — multiply accordingly. Prices change; plug in current ones before promising a number to anyone.
9.2If you can't see it, you can't fix it
An agent decision is invisible unless you record it. The minimum viable observability is a trace per task: every model call with its input token count, every tool call with its arguments and result, every permission decision, and the total cost and duration.
Logs answer "what happened"; metrics answer "how often"; traces answer "why this one". For agents you need all three, plus one thing most people forget: store the exact context sent on a failing call. Without it, reproducing a failure is guesswork, because the prompt is regenerated on every run.
9.3Seven patterns that keep it running
| Pattern | What it prevents | Implementation sketch |
|---|---|---|
| Timeouts everywhere | A hung tool freezing the whole task | Per-tool timeout, plus an end-to-end deadline propagated through the loop |
| Retry with backoff + jitter | Thundering herd; retry storms | 2–3 attempts, exponential delay, jitter, only on transient error codes |
| Idempotency keys | Double-charging, duplicate emails | Client-generated key per intended action; the downstream service dedupes |
| Checkpoints | Losing 40 minutes of work to a crash | Serialise state after each turn; resume from the last good checkpoint |
| Circuit breaker | Hammering a dead dependency | After N failures, fail fast for a cool-down period |
| Fallback model | Provider outage taking you down | Second provider behind the same interface; degrade quality, keep serving |
| Dead-letter queue | Losing tasks that failed permanently | Park the task plus full trace for human review instead of dropping it |
9.4The security lines that matter
- Least-privilege credentials, per task. A token that can read one folder, issued for one job, expiring in an hour. Not the admin key in an environment variable.
- Secrets never enter the context. If the model needs a result, give it the result — not the key that produced it.
- Egress allowlist. An agent that can only reach the four domains it needs cannot exfiltrate your data to a fifth.
- Redact before you send. Strip PII and account numbers before they reach a third-party model, or keep the work on infrastructure you control.
- Audit everything that changes state. Who asked, what was approved, what ran, what changed. This is also your incident review.
- Budget and rate limits per user and per task. Your cost control is also your abuse control.
Costs don't scale linearly with users. They scale with retries, context growth, and the long tail of hard cases. The user who pastes a 200-page contract costs a hundred times the user who asks a question. Cap per-task spend from day one, or your first viral day is also your first invoice.
9.5When a run goes wrong
The first question in an agent incident is not "which model" or "what prompt". It is which turn. A run is a sequence of decisions, and almost every failure is visible in the trace if you read it in order rather than jumping to the error.
| Check | What it rules out | What you are looking for |
|---|---|---|
| Where it first diverged | Everything downstream of the first bad turn | The last turn whose state still matches the goal |
| The exact context sent | Prompt-assembly bugs, stale memory, dropped instructions | Whether the constraint was actually in the prompt at that moment |
| The tool result it saw | Misreading, truncation, silent empty responses | An error string, an empty body, or 200k of text where 2k was expected |
| Permission decisions | Loops caused by denial and retry | The same action attempted repeatedly with the same refusal |
| Cost and latency by span | "It was slow" as a vague complaint | One span holding most of the time or money — usually a retry |
Two patterns cover most real incidents. The first is a silent tool failure: the API returned an error object the model treated as data, so it proceeded confidently from a false premise. The second is a loop that looks like work: the agent keeps taking a plausible action, receiving a refusal or an unchanged result, and trying again. Both are cheap to detect in a trace and nearly invisible in a chat transcript.
Whatever you find, the run should end with one addition: a check that fails next time. An eval case built from the failing trace is worth more than a runbook entry, because it runs on every future change instead of waiting for a human to remember.
Which number should you put on a dashboard first?
60-second recap
- Tokens are the bill — and retries plus growing context are what make it surprising.
- Trace every task: model calls, tool calls, permission decisions, cost, duration, and the exact context of failures.
- Assume failure: timeouts, backoff, idempotency, checkpoints, circuit breakers, fallbacks, dead-letter.
- Least privilege, no secrets in context, egress allowlist, audit trail, per-task budget.