Migrating from TorchServe to Ray Serve Deep Learning Containers on Amazon EKS
- Authors
- Name
- Nino
- Occupation
- Senior Tech Editor
The landscape of machine learning inference infrastructure has shifted rapidly. For years, PyTorch developers relied on TorchServe as the standard production serving engine for PyTorch models. However, with TorchServe transitioning away from active maintenance, infrastructure and MLOps teams are forced to assume full ownership of their GPU inference stack. Managing custom CUDA driver configurations, Python environment dependencies, and distributed serving layers manually introduces massive operational overhead and stability risks.
To address this challenge, AWS introduced pre-tested, fully supported Ray Serve Deep Learning Containers (DLCs). By pairing Ray Serve’s high-performance pythonic serving framework with AWS DLCs on Amazon Elastic Kubernetes Service (Amazon EKS), engineering teams gain a cloud-native, enterprise-grade architecture for large language models (LLMs) and vision-language models (VLMs).
Whether you are scaling in-house GPU clusters or utilizing unified API routers like n1n.ai to orchestrate multi-provider fallback layers, transitioning your self-hosted inference workloads from TorchServe to Ray Serve DLCs is a critical step in modernizing your AI stack.
TorchServe vs. Ray Serve: Architectural Evolution
TorchServe relied heavily on Java-based backend management worker processes coupled with Python worker handlers. While effective for traditional computer vision and small NLP tasks, this architecture struggled with complex dynamic batching, pipeline parallelism, and multi-model composition required by modern generative AI workloads.
Ray Serve, built on top of the distributed computing framework Ray, eliminates the Java abstraction layer entirely. It offers native Python execution, dynamic request routing, actor-based state management, and fine-grained resource allocations (such as fractional GPU assignment).
| Feature | TorchServe | Ray Serve DLC on Amazon EKS |
|---|---|---|
| Runtime Base | Java Frontend + Python Workers | Native Python & Ray Micro-actors |
| Distributed Scaling | Node-level worker management | Cluster-wide auto-scaling with KubeRay |
| Model Composition | Complex custom code chains | Pythonic Directed Acyclic Graphs (DAGs) |
| Container Management | Manual CUDA/Dependencies setup | AWS Pre-tested Deep Learning Containers |
| Multi-Modal Support | Limited dynamic batching for VLMs | High-throughput streaming & dynamic batching |
| Resource Utilization | Coarse-grained per-GPU binding | Fractional GPUs (num_gpus=0.25) per worker |
Why AWS Ray Serve Deep Learning Containers?
Assembling a production-ready GPU container requires aligning compatible versions of Ubuntu, NVIDIA CUDA drivers, cuDNN libraries, PyTorch builds, Ray runtime binaries, and specialized acceleration packages (like vLLM or Hugging Face Transformers). A single mismatched minor version can lead to silent memory leaks or CUDA initialization panics.
AWS Ray Serve DLCs solve this by supplying a pre-validated, optimized docker image layer. They come pre-installed with:
- Verified CUDA and NCCL driver combinations.
- PyTorch binaries compiled for AWS Graviton and x86 GPU instances (e.g., g5, p4de, p5).
- Ray Serve runtime integrated with AWS security, IAM, and CloudWatch telemetry.
- Optimized Triton/vLLM inference engines.
By deploying these containers onto Amazon EKS via the KubeRay Operator, you transform complex GPU orchestration into standard Kubernetes declarative manifests.
Architectural Overview: Deploying a Vision-Language Model (VLM)
The following diagram outlines how client requests flow through an EKS-based Ray Serve cluster using AWS Ray Serve DLCs, alongside how developer workflows compare when hitting external unified model gateways like n1n.ai.
+---------------------------------------+
| Client Application / SDK |
+-------------------++------------------+
|
+------------------+------------------+
| |
[ Internal Direct Endpoint ] [ Global Unified API ]
| |
v v
+------------------------+ +--------------------+
| AWS ALB / Ingress | | n1n.ai |
+-----------+------------+ | Unified Gateway |
| +--------------------+
v |
+----------------------------------+ |
| Amazon EKS Cluster (KubeRay) | |
| +------------------------------+ | v
| | Ray Serve Head Node | | +----------------+
| +--------------+---------------+ | | Hosted LLM |
| | Dynamic Router | | Providers |
| v | +----------------+
| +------------------------------+ |
| | Ray Worker Nodes (NVIDIA GPU)| |
| | - Ray Serve DLC Container | |
| | - VLM Worker Model (Qwen-VL)| |
| +------------------------------+ |
+----------------------------------+
Step-by-Step Implementation Guide
This walk-through demonstrates how to deploy a Vision-Language Model (such as Qwen/Qwen2-VL-7B-Instruct or llava-hf/llava-1.5-7b-hf) on Amazon EKS using the KubeRay operator and Ray Serve DLC.
Step 1: Install the KubeRay Operator
First, add the KubeRay Helm repository and install the operator into your EKS cluster:
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
kubectl create namespace kuberay-system
helm install kuberay-operator kuberay/kuberay-operator --version 1.1.0 -n kuberay-system
Step 2: Define the RayService Kubernetes Manifest
Create a file named vlm_rayservice.yaml. Notice the use of the AWS Ray Serve Deep Learning Container image repository.
apiVersion: ray.io/v1
kind: RayService
metadata:
name: vlm-qwen-service
namespace: default
spec:
serviceUnhealthyThreshold: 300
rayClusterConfig:
rayVersion: '2.35.0'
headGroupSpec:
rayStartParams:
dashboard-host: '0.0.0.0'
template:
spec:
containers:
- name: ray-head
image: 763104351884.dkr.ecr.us-west-2.amazonaws.com/ray-pytorch-inference:2.35.0-gpu-py310-cu121-ubuntu22.04
resources:
limits:
cpu: "4"
memory: "16Gi"
requests:
cpu: "2"
memory: "8Gi"
ports:
- containerPort: 6379
name: gcs
- containerPort: 8265
name: dashboard
- containerPort: 8000
name: serve
workerGroupSpecs:
- groupName: gpu-workers
replicas: 2
minReplicas: 1
maxReplicas: 4
rayStartParams: {}
template:
spec:
containers:
- name: ray-worker
image: 763104351884.dkr.ecr.us-west-2.amazonaws.com/ray-pytorch-inference:2.35.0-gpu-py310-cu121-ubuntu22.04
securityContext:
capabilities:
add: ["SYS_PTRACE"]
resources:
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
requests:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
Step 3: Write the Ray Serve Python Application
Next, build the Python service script (vlm_serve.py) using PyTorch and Hugging Face transformers optimized for multi-modal processing.
import torch
from PIL import Image
import io
import base64
from ray import serve
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration
@serve.deployment(
num_replicas=2,
ray_actor_options=\{"num_gpus": 1, "num_cpus": 4\},
max_ongoing_requests=20
)
class VLMDeployment:
def __init__(self):
model_id = "Qwen/Qwen2-VL-7B-Instruct"
# Load model with bfloat16 for fast GPU inference
self.model = Qwen2VLForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
self.processor = AutoProcessor.from_pretrained(model_id)
print("VLM Model initialized successfully.")
async def __call__(self, request) -> dict:
json_data = await request.json()
prompt = json_data.get("prompt