Building a Multimodal WhatsApp Ordering Assistant with Amazon Bedrock AgentCore

Conversational commerce has evolved beyond basic keyword-driven chatbots. Modern enterprise ordering systems require handling unstructured inputs across multiple modalities—text messages, voice notes, and live voice calls—on a single customer touchpoint like WhatsApp. Achieving this seamless user experience while maintaining state across modalities presents significant architectural challenges.

In this guide, we will explore how to build a production-grade, multimodal WhatsApp ordering assistant using Amazon Bedrock AgentCore and Amazon Nova 2. Additionally, we will demonstrate how integrating unified LLM API gateways such as n1n.ai can help streamline model routing, optimize latency, and provide robust fallbacks across foundation model providers.

Architectural Overview: Decoupled Multimodal Orchestration

A common mistake in building conversational AI is tightly coupling the channel transport layer (WhatsApp API) with the business and ordering logic. To ensure resilience and scalability, the architecture must separate the channel ingest layer, the multimodal transcription engine, the agent execution core, and the persistent memory state.

+-------------------------------------------------------------------------+
|                        WhatsApp Business API                           |
|               (Single Business Phone Number Endpoint)                   |
+-------------------------------------------------------------------------+
                                     |
                                     v
+-------------------------------------------------------------------------+
|                     Webhook Routing Layer (FastAPI)                     |
|        Normalizes payload: Text, Audio Files, WebRTC Media Stream        |
+-------------------------------------------------------------------------+
                                     |
                                     v
+-------------------------------------------------------------------------+
|                    Amazon Bedrock AgentCore & Nova 2                    |
|               (Core Orchestration & Tool Calling Engine)                |
|       Fallback / Multi-Model Support via n1n.ai API Gateway            |
+-------------------------------------------------------------------------+
                                     |
                  +------------------+------------------+
                  |                                     |
                  v                                     v
+-----------------------------------+ +-----------------------------------+
|      Unified Agent Memory Store   | |       Transactional Backends     |
| (DynamoDB Session & Context Sync) | |   (Inventory & Payment Gateway)   |
+-----------------------------------+ +-----------------------------------+

Key Architectural Principles:

Unified Channel Endpoint: Customers interact via a single WhatsApp number regardless of whether they text, send audio notes, or initiate a direct voice call.
Decoupled Business Logic: Bedrock AgentCore manages tool calls (e.g., checking item availability, adding items to cart, processing payments) independently of how input arrived.
Cross-Modal Shared Memory: State (e.g., cart_items, delivery_address, dietary_preferences) persists across interactions. A customer can start an order via text, clarify modifications through a voice note, and finalize delivery details during a quick phone call.

Step-by-Step Implementation Guide

Step 1: Handling Webhook Inputs in Python

The WhatsApp Business API sends distinct JSON payloads based on the incoming media type. Below is an implementation using FastAPI that normalizes inputs before passing them to the Amazon Bedrock AgentCore orchestrator. For high-volume setups requiring reliable LLM routing and failovers, routing model calls through n1n.ai ensures maximum uptime across inference requests.

import os
import httpx
from fastapi import FastAPI, Request, BackgroundTasks
from pydantic import BaseModel
import boto3

app = FastAPI()

# Amazon Bedrock Runtime Client
bedrock_agent = boto3.client('bedrock-agent-runtime', region_name='us-east-1')

# Optional: Initialize n1n.ai client for fallback or specialized model requests
N1N_API_KEY = os.getenv("N1N_API_KEY")
N1N_ENDPOINT = "https://api.n1n.ai/v1/chat/completions"

class WhatsAppMessage(BaseModel):
    phone_number: str
    message_type: str  # 'text', 'audio', or 'call_session'
    content: str       # Text string or S3 URI for audio

async def send_to_bedrock_agent(session_id: str, prompt: str):

Source: https://aws.amazon.com/blogs/machine-learning/deploy-a-multimodal-whatsapp-ordering-assistant-with-amazon-bedrock-agentcore/