Building an Agentic Disaster Recovery Assistant with Amazon Bedrock
- Authors
- Name
- Nino
- Occupation
- Senior Tech Editor
Disaster recovery (DR) at enterprise scale remains one of the most stressful challenges in software engineering. When an AWS region experiences degradation or a core storage layer suffers corruption, on-call engineers are tasked with executing complex, multi-step runbooks under extreme pressure. A single mistyped command during a live regional failover can turn a minor outage into a catastrophic data corruption incident.
To solve this, Intuit developed EWOK Agent (Emergency Workflow Operator Kit), an agentic disaster recovery assistant built on Amazon Bedrock. EWOK allows on-call SREs to initiate, monitor, and validate production failovers using plain-language requests—all while ensuring every operation remains deterministic, fully audited, and strictly policy-compliant.
When evaluating high-availability AI infrastructure to back critical SRE automation workflows, enterprise teams often combine Amazon Bedrock with multi-provider aggregators like n1n.ai to ensure cross-model redundancy and low-latency API access.
The Architecture of EWOK: Bridging LLM Intelligence with Deterministic Safety
The fundamental challenge of using Large Language Models (LLMs) in site reliability engineering is non-determinism. Infrastructure failover requires 100% predictable execution. Intuit bridged this gap by decoupling the LLM's role as a high-level orchestrator and intent parser from the underlying execution and policy evaluation engine.
+-----------------------------------------------------------------------------------+
| User Interface |
| (Slack / CLI / Web Command Interface) |
+-----------------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------------+
| Amazon Bedrock Agent Layer |
| - Intent Extraction (Claude 3.5 Sonnet) |
| - Context & Parameter Resolution |
| - Plan Generation |
+-----------------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------------+
| Deterministic Guardrail & Policy Engine |
| - OPA (Open Policy Agent) Rules |
| - IAM / RBAC Role Verification |
| - Pre-flight Readiness Checks |
+-----------------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------------+
| Execution Layer (Orchestration) |
| - AWS Step Functions / Lambda / Systems Manager |
| - Multi-Region Traffic Switching (Route 53 / ALB) |
+-----------------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------------+
| Telemetry & Audit Pipeline |
| - Immutable Audit Logs (DynamoDB / CloudWatch) |
| - Real-time Slack Status Updates |
+-----------------------------------------------------------------------------------+
Key Components Breakdown:
- Intent Parsing & Reasoning: Powered by Amazon Bedrock (utilizing models such as Anthropic Claude 3.5 Sonnet), the agent parses natural text like "Failover TurboTax tax calculation service from us-east-1 to us-west-2 due to elevated latency."
- Tool / Action Groups: The agent calls predefined AWS Lambda tool hooks mapped to specific infrastructure tasks (e.g., draining connection pools, rerouting DNS, promoting read replicas).
- Policy Guardrails: Before any AWS API is invoked, the proposed actions pass through an independent Open Policy Agent (OPA) layer and Bedrock Guardrails. If the RTO (Recovery Time Objective) or compliance parameters are violated, execution halts instantly.
- Human-in-the-Loop (HITL) Authorization: Destructive operations (e.g., DNS rerouting or active database promotions) require interactive cryptographic authorization from a secondary incident commander.
Step-by-Step Implementation: Building Bedrock Action Groups for DR
To implement an agentic DR assistant, developers configure Amazon Bedrock Agents with explicit JSON schemas for tool calling. Below is an example of an OpenAPI specification for a regional failover tool group, followed by the Python Lambda handler executing the validated action.
1. Defining the OpenAPI Schema for Bedrock Agent
{
"openapi": "3.0.0",
"info": {
"title": "Disaster Recovery Failover API",
"version": "1.0.0"
},
"paths": {
"/failover/execute": {
"post": {
"summary": "Trigger regional service failover",
"description": "Executes traffic shift and database promotion for a target microservice.",
"operationId": "executeFailover",
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"service_name": {
"type": "string",
"description": "The identifier of the service, e.g., turbotax-calc"
},
"source_region": {
"type": "string",
"description": "Current primary region, e.g., us-east-1"
},
"target_region": {
"type": "string",
"description": "Destination region for failover, e.g., us-west-2"
},
"reason": {
"type": "string",
"description": "User-provided rationale for initiating failover"
}
},
"required": ["service_name", "source_region", "target_region", "reason"]
}
}
}
},
"responses": {
"200": {
"description": "Failover workflow successfully initiated"
}
}
}
}
}
}
2. AWS Lambda Action Group Handler with Security Controls
import json
import boto3
import os
step_functions = boto3.client('stepfunctions')
STATE_MACHINE_ARN = os.environ.get("DR_STATE_MACHINE_ARN")
ALLOWED_SERVICES = \{"turbotax-calc