Migrating TorchServe Workloads to Ray Serve Deep Learning Containers on Amazon EKS

The landscape of production machine learning infrastructure is undergoing a fundamental shift. For years, TorchServe stood as the standard open-source framework co-developed by AWS and PyTorch for serving PyTorch models at scale. However, as active maintenance for TorchServe has waned, MLOps and infrastructure teams are left facing a daunting reality: inheriting full ownership of a complex, volatile GPU inference stack. Managing raw CUDA drivers, PyTorch runtime dependencies, C++ compilation layers, and serving frameworks in-house creates massive operational overhead.

To solve this friction, AWS introduced official Ray Serve Deep Learning Containers (DLCs)—pre-tested, highly optimized container images configured with the entire GPU driver stack, Python environment, Ray framework, and underlying serving optimizations like vLLM and FlashAttention. By adopting Ray Serve DLCs, engineering teams can replace custom-built, brittle base images with a fully supported ecosystem.

When architecting scalable AI services, high-availability deployments often combine custom self-hosted microservices on Ray Serve with external API aggregators like n1n.ai to handle peak traffic bursting, fallback routing, and access to proprietary models. In this post, we will walk through migrating from TorchServe to Ray Serve DLCs on Amazon EKS, demonstrating how to deploy a Vision-Language Model (VLM) on a single GPU node.

The Evolution: TorchServe vs. Ray Serve DLCs

TorchServe relied on a Java-based frontend paired with Python worker processes, creating serialization overhead and inter-process IPC bottlenecks. Furthermore, supporting modern LLM and VLM optimizations—such as dynamic token streaming, paged attention, and pipeline parallelism—required complex custom handlers.

Ray Serve, built on top of the distributed Ray Core runtime, uses a pure Python actor framework. This architectural choice aligns seamlessly with the modern AI software stack.

Deep Dive: AWS Ray Serve Deep Learning Container Architecture

The AWS Ray Serve DLC eliminates "dependency hell" by packaging layers that are validated for hardware compatibility on AWS EC2 GPU instances (such as g5, p4d, and p5 families).

+-----------------------------------------------------------------------+
|                       Ray Serve Application Layer                     |
|         (Python FastAPI Endpoint, Model Scaling & Routing Logic)      |
+-----------------------------------------------------------------------+
|                    LLM / VLM Inference Engines                        |
|             (vLLM, Transformers, HuggingFace Accelerate)              |
+-----------------------------------------------------------------------+
|                        Ray Core Distributed Engine                    |
|             (Ray Actors, Task Scheduler, Distributed Plasma Store)    |
+-----------------------------------------------------------------------+
|                   CUDA / cuDNN / NCCL / FlashAttention                |
+-----------------------------------------------------------------------+
|               Base OS (Ubuntu) & PyTorch GPU Runtime                  |
+-----------------------------------------------------------------------+

When building enterprise applications, using pre-validated containers ensures high baseline performance. However, for applications requiring zero downtime and automatic model failover across open-source and proprietary models, integrating an API aggregator like n1n.ai alongside your EKS cluster gives infrastructure teams unified routing across hosted endpoints and managed LLM providers.

Hands-on Tutorial: Deploying a Vision-Language Model on Amazon EKS

Let's walk through deploying a Vision-Language Model (VLM) using the AWS Ray Serve DLC on Amazon EKS equipped with an NVIDIA A10G GPU (g5.xlarge or g5.2xlarge).

Step 1: Cluster Prerequisites and KubeRay Installation

First, ensure you have an active EKS cluster configured with the NVIDIA GPU Operator. Install the KubeRay Operator using Helm:

helm repo add kuberay https://ray-project.github.io/helm-charts/
helm repo update

# Install KubeRay Operator
helm install kuberay-operator kuberay/kuberay-operator --version 1.1.0 \
  --namespace kuberay-operator \
  --create-namespace

Step 2: Write the Ray Serve VLM Application Script

Create a Python script named vlm_serve.py. This script defines an asynchronous Ray Serve deployment that loads a Vision-Language Model (such as Qwen/Qwen2-VL-7B-Instruct or a compact SmolVLM model) using PyTorch and HuggingFace Transformers.

import torch
from PIL import Image
import io
import base64
from fastapi import FastAPI, UploadFile, File, Form
from ray import serve
from transformers import AutoProcessor, AutoModelForVision2Seq

app = FastAPI()

@serve.deployment(
    num_replicas=1,
    ray_actor_options=\{"num_gpus": 1, "num_cpus": 4\}
)
@serve.ingress(app)
class VLMDeployment:
    def __init__(self):
        # Specify model checkpoint
        self.model_id = "HuggingFaceTB/SmolVLM-Instruct"
        print(f"Loading model \{self.model_id\} onto GPU...")
        
        self.processor = AutoProcessor.from_pretrained(self.model_id)
        self.model = AutoModelForVision2Seq.from_pretrained(
            self.model_id,
            torch_dtype=torch.bfloat16,
            device_map="auto"
        )
        print("VLM Model initialized successfully.")

    @app.post("/v1/analyze")
    async def analyze_image(
        self, 
        prompt: str = Form(...), 
        image: UploadFile = File(...)
    ):
        # Read image content asynchronously
        image_bytes = await image.read()
        pil_image = Image.open(io.BytesIO(image_bytes)).convert("RGB")

        # Format input prompt according to model spec
        messages = [
            \{
                "role": "user