# Canary and Rollback

Progressive delivery for AI models: canary deployments, automated rollback, eval gates, and safe rollout patterns.

## Deployment Strategies for AI

| Strategy | Description | Rollback Speed | Risk | Best For |
|---|---|---|---|---|
| **Rolling** | Replace instances gradually | Slow (re-deploy) | Low | Stateless inference, default |
| **Blue-Green** | Two identical environments, switch traffic | Instant | Medium | Critical services, zero-downtime |
| **Canary** | Route small % traffic to new model | Fast (route back) | Low | High-risk changes, model updates |
| **Shadow** | Mirror traffic to new model, discard response | N/A | Zero | Validating new model before serving |
| **Feature Flag** | Deploy code, toggle model exposure | Instant | Lowest | Decouple deploy from release |

## Canary Deployment for Models

```mermaid
stateDiagram-v2
    [*] --> Canary: deploy new model, route 5% traffic
    Canary --> Production: pass eval gate + metrics healthy
    Canary --> Rollback: eval failure or SLO breach
    Production --> Canary: next model version
    Rollback --> [*]: route 100% back to previous
    
    note right of Canary: Automated metric analysis decides promote/rollback
```

```yaml
# Canary deployment configuration
canary:
  model: customer-support-llm:v4
  baseline_model: customer-support-llm:v3
  
  # Traffic split
  initial_traffic_percent: 5
  max_traffic_percent: 100
  traffic_increment_step: 15
  increment_interval_min: 30
  
  # Eval gate (must pass before any promotion)
  eval_gate:
    enabled: true
    judge_model: gpt-4o-mini
    sample_rate: 0.1
    thresholds:
      faithfulness: 0.90
      relevance: 0.85
      safety: 1.00
  
  # SLOs (automatic rollback if breached)
  slos:
    p99_latency_ms: 2000
    error_rate: 0.01
    cost_per_1k_tokens_usd: 0.0015
    hallucination_rate: 0.02
  
  # Bake period before promoting
  bake_period_min: 60
  
  # Rollback trigger
  rollback_on:
    - eval_gate_failure
    - slo_breach
    - guardrail_violation_rate > 0.05
```

## Automated Rollback Logic

```python
from enum import Enum
from dataclasses import dataclass
from typing import Optional

class DeploymentState(Enum):
    CANARY = "canary"
    PRODUCTION = "production"
    ROLLBACK = "rollback"

@dataclass
class SLOMetrics:
    p99_latency_ms: float
    error_rate: float
    cost_per_1k_tokens_usd: float
    hallucination_rate: float
    guardrail_violation_rate: float

@dataclass
class CanaryDecision:
    action: str  # "promote", "rollback", "continue"
    reason: str
    confidence: float

class CanaryController:
    def __init__(self, slos: dict, eval_gate):
        self.slos = slos
        self.eval_gate = eval_gate
        self.baseline_metrics = None
    
    def evaluate(self, current: SLOMetrics, eval_result: dict) -> CanaryDecision:
        # 1. Check eval gate
        if not eval_result["passed"]:
            return CanaryDecision(
                action="rollback",
                reason=f"Eval gate failed: {eval_result['failures']}",
                confidence=1.0,
            )
        
        # 2. Check SLOs against baseline
        violations = []
        if current.p99_latency_ms > self.baseline_metrics.p99_latency_ms * 1.2:
            violations.append("p99_latency")
        if current.error_rate > self.slos["error_rate"]:
            violations.append("error_rate")
        if current.hallucination_rate > self.slos["hallucination_rate"]:
            violations.append("hallucination_rate")
        
        if violations:
            return CanaryDecision(
                action="rollback",
                reason=f"SLO violations: {', '.join(violations)}",
                confidence=0.95,
            )
        
        # 3. All checks passed — continue or promote
        return CanaryDecision(
            action="continue",
            reason="All SLOs and eval gate passing",
            confidence=1.0,
        )
```

## Rollback Playbook

| Trigger | Detection | Rollback Action | Owner |
|---|---|---|---|
| Eval gate failure | Automated eval on canary traffic | Route 100% to previous model version | ML Engineer |
| SLO breach (latency/error) | Observability alert | Instant route back to baseline | SRE |
| Hallucination spike | Output guardrail alerts | Rollback + investigate training data | ML + Security |
| Cost spike | Cost per token alert | Route to cheaper fallback model | Platform Eng |
| Guardrail violation rate | Real-time guardrail logs | Block new model, rollback | Security |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Deploying models without eval gate | Add evaluation before any Production promotion |
| No automated rollback | Every canary must have automatic rollback on SLO breach |
| Rollback requiring redeploy | Keep previous model version hot; rollback = config change |
| No bake period | Wait for delayed-onset issues before promoting |
| Canary too large (50%+ start) | Start at 1–5%; increase gradually |

## Related Notes

- [[MLOps/Model-Registry-And-Lineage.md]] — Model stages, champion/challenger
- [[Observability/Eval-In-Production.md]] — Continuous evaluation, LLM-as-judge
- [[AgenticEngineering/Production/Deployment-Architecture.md]] — Deployment patterns
- [[AgenticEngineering/Production/Reliability.md]] — SLOs, error budgets, circuit breakers
- [[Reliability/Guardrails-And-Safety.md]] — Guardrails, incident response

## Progressive Delivery with Feature Flags

```python
from launchdarkly import LDClient

ld_client = LDClient("sdk-key")

def get_model_for_user(user_context):
    # Feature flag controls which model serves which users
    if ld_client.variation("new-llm-model", user_context, False):
        return "customer-support-llm:v4"  # canary
    return "customer-support-llm:v3"      # baseline

# Gradually increase canary percentage
def canary_rollout(percentage: int):
    ld_client.track("canary-rollout", {"traffic_percent": percentage})
    # Update targeting rule to route `percentage` of users to new model
```

## Sources

* Progressive delivery (LaunchDarkly, Argo Rollouts, Flagger), Canary deployments (Google SRE, Istio), ML model deployment patterns, AWS CodeDeploy, Spinnaker, ArgoCD.*