Human-in-the-Loop Patterns

Designing meaningful human control and collaboration in agent systems.

HITL Philosophy

Goal: Not "human in the loop" as bottleneck, but human on the loop — supervisory, strategic, exception-handling.

Autonomy Spectrum

Fully Autonomous ←────────────────────────────────────→ Fully Human
      │                                                      │
      ▼                                                      ▼
Agent decides                                          Human decides
all actions                                              every action
      │                                                      │
      ▼                                                      ▼
Human: "Here's the goal"                          Human: "Click this"
Agent: Plans, executes,                           Agent: None
      reports                                          (or tool executor)

Design Principle

Default to autonomy, escalate on uncertainty. - Agent handles 95%+ routine decisions - Human handles: high stakes, ambiguity, policy, ethics, novel situations

HITL Patterns

1. Pre-Action Approval (Gate)

async def execute_with_approval(action, context):
    if action.risk_level >= HIGH:
        approval = await hitl.request(
            f"Approve {action.name}?",
            context=context,
            options=["Approve", "Modify", "Reject"],
            timeout=300
        )
        if approval != "Approve":
            raise HumanRejected(approval)
    return await action.execute()

Use for: Destructive actions, financial transactions, external communications, policy violations.

2. Plan Approval

async def approve_plan(plan, goal):
    # Present plan with rationale
    presentation = PlanPresentation(
        goal=goal,
        steps=plan.steps,
        estimated_cost=plan.estimated_cost,
        estimated_time=plan.estimated_time,
        risks=plan.identified_risks,
        alternatives=plan.alternatives
    )
    return await hitl.request_approval(presentation)

Use for: Complex multi-step workflows, costly operations, irreversible sequences.

3. Post-Action Review

async def execute_with_review(action, context):
    result = await action.execute()
    if action.requires_review:
        review = await hitl.review(
            action=action,
            result=result,
            criteria=["accuracy", "completeness", "policy_compliance"]
        )
        if review.action == "rollback":
            await action.rollback()
        elif review.action == "modify":
            result = await apply_modification(result, review.changes)
    return result

Use for: Content generation, code changes, customer-facing output.

4. Escalation on Uncertainty

async def escalate_on_uncertainty(agent, task):
    result = await agent.run(task)
    if result.confidence < THRESHOLD or result.flagged:
        escalation = Escalation(
            reason="low_confidence" | "policy_ambiguity" | "novel_situation",
            context=task,
            agent_state=agent.get_state(),
            suggested_actions=["clarify_goal", "provide_example", "take_over"]
        )
        return await human.handle_escalation(escalation)
    return result

Use for: Edge cases, novel situations, confidence calibration.

5. Continuous Supervision (Human-on-the-Loop)

class Supervisor:
    def __init__(self):
        self.dashboard = LiveDashboard()
        self.alerts = AlertManager()
        self.interventions = InterventionLog()

    async def monitor(self, agent):
        async for event in agent.stream_events():
            self.dashboard.update(event)

            if event.type == "high_risk_action":
                self.alerts.notify_supervisor(event)

            if event.type == "anomaly_detected":
                intervention = await self.request_intervention(event)
                if intervention:
                    agent.intervene(intervention)

Use for: Long-running autonomous agents, production systems, safety-critical.

HITL Interface Design

For Humans (Dashboards)

For Agents (API)

class HITLInterface:
    async def request_approval(self, request: ApprovalRequest) -> ApprovalResponse:
        """Blocking call with timeout"""

    async def request_input(self, prompt: str, schema: JSONSchema) -> dict:
        """Structured input from human"""

    async def notify(self, notification: Notification) -> None:
        """Fire-and-forget"""

    async def escalate(self, escalation: Escalation) -> Resolution:
        """Blocking with callback"""

    def stream_events(self) -> AsyncIterator[Event]:
        """For supervisor dashboard"""

Approval UX Patterns

Slack/Teams Integration

# Slack Block Kit example
approval_block = {
    "type": "section",
    "text": {"type": "mrkdwn", "text": f"*Action Required*\n{action.description}"},
    "accessory": {
        "type": "button",
        "text": {"type": "plain_text", "text": "Approve"},
        "style": "primary",
        "value": f"approve:{action.id}"
    }
}

Dashboard Modal

// React component
<ApprovalModal
  title="Deploy to Production?"
  details={{
    environment: "prod",
    changes: ["User auth refactor", "Rate limit increase"],
    riskScore: 0.7,
    estimatedImpact: "5min downtime"
  }}
  onApprove={() => hitl.approve(requestId)}
  onReject={() => hitl.reject(requestId)}
  onModify={(changes) => hitl.modify(requestId, changes)}
/>

CLI/TUI

$ agent run deploy --env prod
⚠ High-risk action: Deploy to production
   Changes: User auth refactor, Rate limit increase
   Risk score: 0.7/1.0
   Est. downtime: 5 minutes

   [A]pprove  [M]odify  [R]eject  [V]iew details  [?] Help
> A
✓ Approved. Deploying...

HITL for Specific Domains

Coding Agents

Customer-Facing Agents

Research Agents

HITL Metrics

Anti-Patterns

Implementation Checklist

• [ ] Risk matrix defined (what triggers HITL)
• [ ] Approval UI built (Slack, dashboard, CLI, email)
• [ ] Timeout + default action for every gate
• [ ] Structured context for human decisions
• [ ] Audit log of all HITL interactions
• [ ] Metrics dashboard (intervention rate, latency)
• [ ] Escalation path for human uncertainty
• [ ] Regular review of HITL thresholds
• [ ] Human feedback → agent improvement loop

Related Notes

• Control Flow — HITL as control flow gate
• Single Agent Patterns — ReAct with HITL
• LangGraph — interrupt() for HITL
• OpenAI Agents SDK — Guardrails, approvals
• Observability — Human-facing dashboards
• Security — Approval for sensitive actions
• Coding Agents — PR approval gates
• Customer Facing Agents — Escalation patterns
• Red Teaming — Human red-teamers

Sources: LangGraph interrupt(), OpenAI Agents SDK guardrails, AutoGen UserProxyAgent, CrewAI human tools, Microsoft Responsible AI, Google PAIR, HCI literature on human-AI interaction, "Human-in-the-Loop" surveys (Wu et al., 2022), production post-mortems from Cognition, Cursor, GitHub Copilot.