Deploying OpenAI Codex with LiteLLM Gateway on Amazon ECS and Amazon Bedrock

As modern software engineering teams integrate AI coding assistants into their daily workflows, governing model access, managing API costs, and tracking telemetry have become paramount concerns. Deploying an enterprise-grade OpenAI-compatible gateway gives platform teams centralized control while preserving the developer experience across IDEs, CLI tools, and automated pipelines.

In this guide, we walk through deploying a customer-operated LiteLLM Gateway on Amazon ECS (AWS Fargate), connecting it to models hosted on Amazon Bedrock, and configuring coding tools like OpenAI Codex to route through scoped identity endpoints. We will also compare this architecture against direct AWS IAM Identity Center authentication and a managed SaaS provider like Portkey.

Architectural Overview & Governance Paradigms

When providing developers access to foundational models (FMs) on cloud platforms like AWS, organizations typically evaluate three architectural patterns:

1. Direct AWS IAM / Bedrock Access: Developers authenticate via AWS credentials directly against Amazon Bedrock APIs.
2. Self-Hosted Proxy (LiteLLM on ECS Fargate): A lightweight, open-source proxy layer deployed inside your AWS VPC that abstracts provider specifics, enforces virtual API keys, caps individual team budgets, and exposes standard OpenAI-compatible REST endpoints.
3. Managed Gateway SaaS (e.g., Portkey / Enterprise Hubs): Fully managed control planes handling routing, caching, and analytics across multi-cloud infrastructure.

For enterprise teams requiring high resilience and multi-provider failover without cloud vendor lock-in, hybrid orchestration platforms like n1n.ai provide a streamlined approach to accessing unified LLM endpoints across global infrastructure.

Architectural Comparison

Infrastructure Prerequisites

Before launching the LiteLLM gateway on Amazon ECS, ensure your environment satisfies the following components:

• An AWS Account with access to Amazon Bedrock model inference (e.g., Anthropic Claude 3.5 Sonnet, Amazon Nova, or OpenAI models available via Bedrock routing).
• An Amazon ECS Cluster using AWS Fargate serverless compute.
• An Amazon ElastiCache Redis cluster (or AWS Serverless Redis) for distributed rate-limiting and temporary state caching.
• An Amazon Aurora PostgreSQL or RDS PostgreSQL database to maintain LiteLLM user identities, virtual keys, and transactional audit logs.

Step 1: Building the LiteLLM Gateway Configuration

LiteLLM relies on a declarative config.yaml file to map incoming request routes (such as standard OpenAI model strings) to backend Bedrock model IDs.

Create the following config.yaml file to establish your model routing layer:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
      aws_region_name: us-east-1
      max_tokens: 4096

  - model_name: codex
    litellm_params:
      model: bedrock/anthropic.claude-3-haiku-20240307-v1:0
      aws_region_name: us-east-1

  - model_name: text-embedding-3-small
    litellm_params:
      model: bedrock/amazon.titan-embed-text-v2:0
      aws_region_name: us-east-1

router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: "litellm-redis.internal.vpc.amazonaws.com"
  redis_port: 6379

general_settings:
  master_key: "os.environ/LITELLM_MASTER_KEY"
  database_url: "os.environ/DATABASE_URL"

Note: Mapping gpt-4o and codex aliases directly to Amazon Bedrock endpoints enables local developer tools configured for standard OpenAI API endpoints to seamlessly execute inference against Bedrock models.

Step 2: Deploying to AWS Fargate with Terraform

Next, package LiteLLM into an AWS Fargate container task. The task execution role requires permissions to query AWS Secrets Manager and invoke Bedrock models.

Below is an abbreviated HCL snippet illustrating the AWS ECS Task Definition and IAM policy setup:

resource "aws_iam_role