Building an Agentic Disaster Recovery Assistant with Amazon Bedrock: Lessons from Intuit
- Authors
- Name
- Nino
- Occupation
- Senior Tech Editor
Modern enterprise applications operate at a scale where downtime translates directly into millions of dollars in lost revenue and severe regulatory penalties. For financial technology powerhouses like Intuit—the company behind TurboTax, QuickBooks, and Credit Karma—disaster recovery (DR) is not merely an IT checklist item; it is a mission-critical imperative.
Executing a production multi-region failover under high-stress incident scenarios presents significant engineering challenges. Traditional DR relies heavily on extensive runbooks, complex shell scripts, manual verification across distributed monitoring dashboards, and tedious human orchestration. When an outage hits, on-call Site Reliability Engineers (SREs) must parse thousands of lines of documentation, evaluate real-time telemetry, and manually trigger infrastructure changes. This human bottleneck increases Mean Time to Resolution (MTTR) and introduces human error risks during critical moments.
To solve this challenge, Intuit engineered EWOK Agent (Emergency Workflow Operations Kit), an autonomous disaster recovery assistant built on Amazon Bedrock. EWOK Agent allows SREs and on-call engineers to initiate, track, and complete complex multi-region failovers using plain-language requests—all while maintaining policy compliance, auditable execution, and strict safety guardrails.
The Evolution of Disaster Recovery: From Static Runbooks to Agentic AI
Disaster recovery systems have evolved through three distinct paradigms:
- Manual Static Runbooks: Documentation-heavy procedures where engineers read instructions from wikis and manually execute CLI commands. High latency, error-prone, and rapidly outdated.
- Scripted ChatOps: Rule-based Slack/Teams bots triggering pre-written Jenkins or AWS Systems Manager pipelines. Faster execution, but rigid and unable to dynamically adapt to unexpected systemic failures.
- Agentic AI Orchestration: Dynamic, context-aware autonomous agents powered by Large Language Models (LLMs). Agents interpret intent, evaluate telemetry, reason through execution plans, invoke infrastructure APIs, and verify system state autonomously under human supervision.
Intuit chose Amazon Bedrock as the underlying foundation for EWOK Agent due to its enterprise-grade security, access to top-tier foundation models like Anthropic's Claude 3.5 Sonnet, native support for Action Groups, Knowledge Bases, and strict Guardrails.
Deep Dive: EWOK Agent Architectural Blueprint
The architecture of an agentic DR assistant must satisfy three foundational principles: Determinism in Action, Contextual Intelligence, and Zero-Trust Security.
Below is a breakdown of how EWOK Agent integrates with AWS infrastructure and Amazon Bedrock:
+-----------------------------------+
| On-Call Engineer (Slack/CLI) |
+-----------------------------------+
|
[Natural Language Request]
v
+-----------------------------------+
| Intuit API Gateway / Router |
+-----------------------------------+
|
v
+-----------------------------------------------------------------------------------+
| Amazon Bedrock Agent Runtime |
| |
| +------------------------+ +-----------------------+ +------------------+ |
| | Bedrock Guardrails | | LLM Reasoning Engine | | Knowledge Base | |
| | (Policy & Safety Filter| -> | (Claude 3.5 Sonnet) | -> | (DR Runbooks, | |
| +------------------------+ +-----------------------+ | Topology Specs) | |
| | +------------------+ |
| v |
| +-----------------------+ |
| | Action Group Dispatch | |
| +-----------------------+ |
+--------------------------------------------|--------------------------------------+
|
v
+-----------------------------------+
| Human-in-the-Loop (HITL) Gate |
| (Approval Prompt & Impact Matrix) |
+-----------------------------------+
|
[Approved Action]
v
+-----------------------------------+
| AWS Lambda / Execution Microservices|
+-----------------------------------+
|
+-------------------------+-------------------------+
| |
v v
+---------------------------+ +---------------------------+
| Amazon Route 53 / Global | | Kubernetes Clusters (EKS) |
| Traffic Management | | Active-Active Failover |
+---------------------------+ +---------------------------+
Key Architecture Components
- Reasoning & Planning Engine: Powered by Anthropic's Claude 3.5 Sonnet via Amazon Bedrock. The model breaks complex user requests (e.g., "Shift 50% of payment service traffic from us-east-1 to us-west-2 due to elevated latency") into sequential execution steps.
- Action Groups & OpenAPI Specs: Bedrock Action Groups translate LLM intent into structured API calls. Intuit defines OpenAPI schemas for infrastructure tasks such as modifying DNS weights in Amazon Route 53, scaling AWS EKS deployment replicas, or rerouting Kafka message streams.
- Knowledge Bases (RAG): Amazon Bedrock Knowledge Bases index Intuit's architecture diagrams, dependency trees, and DR runbooks using vector embeddings. This ensures the LLM understands service topology before making planning decisions.
- Policy Engine & Guardrails: Bedrock Guardrails scan incoming queries and generated outputs to block unauthorized prompts, prevent hallucinated API parameters, and enforce strict PII/PHI data masking.
Step-by-Step Implementation: Building a Bedrock DR Action Handler
To illustrate how EWOK Agent executes infrastructure operations safely, consider the following Python implementation using the AWS SDK (boto3) and an Amazon Bedrock Action Group Lambda handler.
1. Defining the Action Group Lambda Endpoint
When Bedrock decides to trigger a failover action, it passes structured JSON events to a target AWS Lambda function.
import json
import boto3
import os
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
route53_client = boto3.client('route53')
# Pre-defined policy limits for safety
MAX_ALLOWED_TRAFFIC_SHIFT_PERCENT = 100
CRITICAL_SERVICES = ["payment-gateway