Starphix
Systems Engineering / Sources
The Lab / Sources / Systems Engineering / Reliability/Guardrails-And-Safety.md

Guardrails and Safety

Input and output guardrails, failover patterns, and incident response for AI systems operating in production.

Guardrail Types

Type Position Purpose Examples
Input Guardrail Before model Block unsafe prompts, PII, jailbreak attempts Profanity filter, PII detector, prompt injection guard
Output Guardrail After model Block unsafe or off-topic responses Toxicity filter, hallucination check, format validator
Semantic Guardrail Before/after Detect off-topic or malicious intent Embedding similarity to allowed topics
Rate Limit Guardrail Before model Prevent abuse and cost spikes Requests per user per minute, token budget per day

Guardrail Architecture

graph TD
    A[User Input] --> B[Input Guardrails]
    B -->|Blocked| C[Reject / Sanitize]
    B -->|Pass| D[Model Inference]
    D --> E[Output Guardrails]
    E -->|Blocked| F[Fallback Response]
    E -->|Pass| G[User Response]

    B --> B1[PII Detection]
    B --> B2[Jailbreak Detection]
    B --> B3[Topic Filter]

    E --> E1[Toxicity Check]
    E --> E2[Hallucination Detection]
    E --> E3[Format Validation]

    style C fill:#ffcdd2
    style F fill:#fff3e0
    style G fill:#e8f5e9

Decision Guidance

Situation Recommended Approach
User-facing chatbot Input guardrails (PII, jailbreak) + output guardrails (toxicity, hallucination)
Safety-critical system Fallback model + human-in-the-loop for high-stakes outputs
High cost per request Circuit breaker on expensive models; route to cheaper fallback
Model degradation Error budget triggers rollback to previous model version

Anti-Patterns

Anti-Pattern Fix
No guardrails on user input Add PII and jailbreak detection before the model sees input
No fallback when model fails Route to a smaller model or cached response on failure
Retrying forever on bad outputs Circuit breaker; stop calling a consistently failing model
Deploying without an incident playbook Write AI-specific runbooks: bad output, data leak, cost spike
Silent guardrail failures Log every guardrail decision; alert on block rates
  • Circuit Breakers And Fallbacks — Circuit breakers, fallback models, error budgets
  • Reliability — Retries, fallbacks, circuit breakers, SLOs
  • Security — Auth, authorization, data privacy, PII
  • Observability — Monitoring and alerting for reliability
  • MLOps — Rollback and deployment safety

Guardrail Example: NVIDIA NeMo Guardrails

from nemoguardrails import LLMRails
from nemoguardrails.actions import action

# Configure guardrails
config = {
    "models": [
        {"type": "main", "engine": "openai", "model": "gpt-4o-mini"},
    ],
    "rails": {
        "input": [
            {"definition": "block user messages containing PII"},
            {"definition": "block jailbreak attempts"},
        ],
        "output": [
            {"definition": "block responses with toxic language"},
            {"definition": "block responses containing unverified medical advice"},
        ],
    },
}

rails = LLMRails(config)

# Register custom action for PII detection
@action(is_system_action=True)
async def detect_pii(text: str):
    # PII detection logic here
    return {"contains_pii": False, "entities": []}

# Generate with guardrails
response = await rails.generate_async(
    messages=[{"role": "user", "content": "Hello, how can you help me today?"}]
)
print(response["content"])

Incident Response Playbook

Incident Detection Response Owner
Hallucinated output Output guardrail + user report Trigger fallback model, alert on-call ML Engineer
Data leak in response Input/output PII guardrail Block response, revoke tokens, investigate Security + ML
Cost spike Cost per token alert Circuit breaker on expensive model, route to cheaper Platform Eng
Model unresponsive Latency SLO breach Failover to secondary model/region SRE

Sources

  • NVIDIA NeMo Guardrails, LlamaGuard (Meta), Microsoft Guidance, AWS Bedrock Guardrails, Google SRE, AI incident response playbooks, Purple Llama.*

From the Systems Engineering research vault · CC BY 4.0 · all notes · raw markdown