Canary and Rollback

Progressive delivery for AI models: canary deployments, automated rollback, eval gates, and safe rollout patterns.

Deployment Strategies for AI

Strategy	Description	Rollback Speed	Risk	Best For
Rolling	Replace instances gradually	Slow (re-deploy)	Low	Stateless inference, default
Blue-Green	Two identical environments, switch traffic	Instant	Medium	Critical services, zero-downtime
Canary	Route small % traffic to new model	Fast (route back)	Low	High-risk changes, model updates
Shadow	Mirror traffic to new model, discard response	N/A	Zero	Validating new model before serving
Feature Flag	Deploy code, toggle model exposure	Instant	Lowest	Decouple deploy from release

Canary Deployment for Models

stateDiagram-v2
    [*] --> Canary: deploy new model, route 5% traffic
    Canary --> Production: pass eval gate + metrics healthy
    Canary --> Rollback: eval failure or SLO breach
    Production --> Canary: next model version
    Rollback --> [*]: route 100% back to previous

    note right of Canary: Automated metric analysis decides promote/rollback

# Canary deployment configuration
canary:
  model: customer-support-llm:v4
  baseline_model: customer-support-llm:v3

  # Traffic split
  initial_traffic_percent: 5
  max_traffic_percent: 100
  traffic_increment_step: 15
  increment_interval_min: 30

  # Eval gate (must pass before any promotion)
  eval_gate:
    enabled: true
    judge_model: gpt-4o-mini
    sample_rate: 0.1
    thresholds:
      faithfulness: 0.90
      relevance: 0.85
      safety: 1.00

  # SLOs (automatic rollback if breached)
  slos:
    p99_latency_ms: 2000
    error_rate: 0.01
    cost_per_1k_tokens_usd: 0.0015
    hallucination_rate: 0.02

  # Bake period before promoting
  bake_period_min: 60

  # Rollback trigger
  rollback_on:
    - eval_gate_failure
    - slo_breach
    - guardrail_violation_rate > 0.05

Automated Rollback Logic

from enum import Enum
from dataclasses import dataclass
from typing import Optional

class DeploymentState(Enum):
    CANARY = "canary"
    PRODUCTION = "production"
    ROLLBACK = "rollback"

@dataclass
class SLOMetrics:
    p99_latency_ms: float
    error_rate: float
    cost_per_1k_tokens_usd: float
    hallucination_rate: float
    guardrail_violation_rate: float

@dataclass
class CanaryDecision:
    action: str  # "promote", "rollback", "continue"
    reason: str
    confidence: float

class CanaryController:
    def __init__(self, slos: dict, eval_gate):
        self.slos = slos
        self.eval_gate = eval_gate
        self.baseline_metrics = None

    def evaluate(self, current: SLOMetrics, eval_result: dict) -> CanaryDecision:
        # 1. Check eval gate
        if not eval_result["passed"]:
            return CanaryDecision(
                action="rollback",
                reason=f"Eval gate failed: {eval_result['failures']}",
                confidence=1.0,
            )

        # 2. Check SLOs against baseline
        violations = []
        if current.p99_latency_ms > self.baseline_metrics.p99_latency_ms * 1.2:
            violations.append("p99_latency")
        if current.error_rate > self.slos["error_rate"]:
            violations.append("error_rate")
        if current.hallucination_rate > self.slos["hallucination_rate"]:
            violations.append("hallucination_rate")

        if violations:
            return CanaryDecision(
                action="rollback",
                reason=f"SLO violations: {', '.join(violations)}",
                confidence=0.95,
            )

        # 3. All checks passed — continue or promote
        return CanaryDecision(
            action="continue",
            reason="All SLOs and eval gate passing",
            confidence=1.0,
        )

Rollback Playbook

Trigger	Detection	Rollback Action	Owner
Eval gate failure	Automated eval on canary traffic	Route 100% to previous model version	ML Engineer
SLO breach (latency/error)	Observability alert	Instant route back to baseline	SRE
Hallucination spike	Output guardrail alerts	Rollback + investigate training data	ML + Security
Cost spike	Cost per token alert	Route to cheaper fallback model	Platform Eng
Guardrail violation rate	Real-time guardrail logs	Block new model, rollback	Security

Anti-Patterns

Anti-Pattern	Fix
Deploying models without eval gate	Add evaluation before any Production promotion
No automated rollback	Every canary must have automatic rollback on SLO breach
Rollback requiring redeploy	Keep previous model version hot; rollback = config change
No bake period	Wait for delayed-onset issues before promoting
Canary too large (50%+ start)	Start at 1–5%; increase gradually

Model Registry And Lineage — Model stages, champion/challenger
Eval In Production — Continuous evaluation, LLM-as-judge
Deployment Architecture — Deployment patterns
Reliability — SLOs, error budgets, circuit breakers
Guardrails And Safety — Guardrails, incident response

Progressive Delivery with Feature Flags

from launchdarkly import LDClient

ld_client = LDClient("sdk-key")

def get_model_for_user(user_context):
    # Feature flag controls which model serves which users
    if ld_client.variation("new-llm-model", user_context, False):
        return "customer-support-llm:v4"  # canary
    return "customer-support-llm:v3"      # baseline

# Gradually increase canary percentage
def canary_rollout(percentage: int):
    ld_client.track("canary-rollout", {"traffic_percent": percentage})
    # Update targeting rule to route `percentage` of users to new model

Sources

Progressive delivery (LaunchDarkly, Argo Rollouts, Flagger), Canary deployments (Google SRE, Istio), ML model deployment patterns, AWS CodeDeploy, Spinnaker, ArgoCD.*

From the Systems Engineering research vault · CC BY 4.0 · all notes · raw markdown

Canary and Rollback

Deployment Strategies for AI

Canary Deployment for Models

Automated Rollback Logic

Rollback Playbook

Anti-Patterns

Related Notes

Progressive Delivery with Feature Flags

Sources