Run multi-node inference workloads using Ray LLM Deep Learning Containers

Deploying massive large language models like DeepSeek-V3 or Claude 3.5 Sonnet across multiple GPU nodes presents significant engineering challenges. Traditional single-node serving frameworks often hit memory walls when handling models with hundreds of billions of parameters. To overcome this, enterprise developers are turning to distributed architectures that separate prefill and decode phases. By leveraging Amazon EKS and pre-built Ray LLM Deep Learning Containers, engineering teams can now orchestrate high-throughput, low-latency multi-node inference clusters without manually compiling complex software stacks from scratch.

The Architecture of Distributed LLM Serving

When scaling inference workloads, running a single model instance on a single GPU or even a single multi-GPU node is frequently insufficient for production SLAs. The compute characteristics of large language models diverge sharply between the prefill phase (processing the input prompt in parallel) and the decode phase (generating tokens autoregressively, one by one).

Prefill operations are compute-bound and benefit immensely from massive parallel processing across multiple GPUs. Conversely, decode operations are memory-bandwidth-bound, requiring optimal KV-cache management and fast interconnects. By decoupling these two stages across distinct node pools—dedicated prefill nodes and dedicated decode nodes—enterprises can maximize hardware utilization and achieve predictable latency profiles.

To bridge this distributed infrastructure seamlessly, you need a unified API gateway that abstracts the complexity. This is where modern aggregation layers and routing strategies come into play. For applications that require high availability and instantaneous fallback mechanisms across multiple foundational models, developers frequently integrate their custom cluster endpoints with advanced LLM API aggregators like n1n.ai to ensure zero-downtime operations.

Prerequisites and Environment Setup

Before deploying your multi-node Ray cluster on Amazon EKS, ensure your AWS environment meets the following requirements:

1. An active Amazon EKS cluster running Kubernetes version 1.28 or higher.
2. Node groups equipped with high-performance GPUs (such as p4de.24xlarge or p5.48xlarge instances) featuring high-speed Elastic Fabric Adapter (EFA) networking.
3. The AWS CLI, kubectl, and Helm configured with appropriate cluster administrator permissions.
4. An active account to pull the official Ray LLM Deep Learning Containers from the AWS Deep Learning Container registry.

Step-by-Step Deployment Guide

Step 1: Configuring the Ray Cluster Head Node

The Ray cluster head node coordinates task scheduling, resource allocation, and worker registration. Create a Kubernetes deployment manifest named ray-head.yaml utilizing the pre-built Ray LLM Deep Learning Container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ray-head
  namespace: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      ray.io/node-type: head
  template:
    metadata:
      labels:
        ray.io/node-type: head
    spec:
      containers:
      - name: ray-head
        image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-ray-inference:2.4.0-gpu-py310
        resources:
          limits:
            cpu: "16"
            memory: "64Gi"
            nvidia.com/gpu: "0"
          requests:
            cpu: "8"
            memory: "32Gi"
        ports:
        - containerPort: 6379
          name: gcs
        - containerPort: 8265
          name: dashboard
        - containerPort: 8000
          name: serve
        command: ["/bin/bash", "-c", "ray start --head --port=6379 --dashboard-host=0.0.0.0 --block"]

Apply the manifest to your cluster:

kubectl apply -f ray-head.yaml -n llm-inference

Step 2: Deploying Worker Nodes with EFA Support

Next, deploy the worker nodes that will host the distributed model weights across multiple GPUs. These worker pods utilize Ray's backend to communicate efficiently via NCCL over EFA interfaces. Create ray-worker.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ray-worker
  namespace: llm-inference
spec:
  serviceName: ray-worker
  replicas: 2
  selector:
    matchLabels:
      ray.io/node-type: worker
  template:
    metadata:
      labels:
        ray.io/node-type: worker
    spec:
      containers:
      - name: ray-worker
        image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-ray-inference:2.4.0-gpu-py310
        securityContext:
          capabilities:
            add: ["IPC_LOCK"]
        resources:
          limits:
            cpu: "32"
            memory: "256Gi"
            nvidia.com/gpu: "8"
          requests:
            cpu: "16"
            memory: "128Gi"
            nvidia.com/gpu: "8"
        command: 
          - "/bin/bash"
          - "-c"
          - >-
            ray start --address=ray-head.llm-inference.svc.cluster.local:6379 --block

Deploy the workers:

kubectl apply -f ray-worker.yaml -n llm-inference

Step 3: Configuring Prefill and Decode Separation

To achieve optimal disaggregated serving, configure your Ray Serve application configuration file (serve-config.yaml) to route prompt ingestion requests to the prefill node pool and token generation requests to the decode node pool:

proxy_location: EveryNode
http_options:
  host: 0.0.0.0
  port: 8000
applications:
  - name: disaggregated_llm_service
    route_prefix: /v1
    import_path: ray_llm.serve:Deployment
    runtime_env:
      pip:
        - vllm==0.5.4
    args:
      model_id: "meta-llama/Meta-Llama-3-70B-Instruct"
      tensor_parallel_size: 8
      pipeline_parallel_size: 2
      prefill_node_selector:
        ray.io/node-type: "prefill-worker"
      decode_node_selector:
        ray.io/node-type: "decode-worker"

Interfacing with OpenAI-Compatible Endpoints

Once your Ray Serve deployment is up and running, it exposes a native OpenAI-compatible REST API. You can test your multi-node inference cluster using standard Python client libraries or curl commands. For developers seeking to benchmark their internal cluster performance against commercial APIs or looking for managed fallback solutions, routing requests through an aggregator like n1n.ai allows seamless switching between local EKS endpoints and cloud-hosted LLMs.

Here is how you can query your newly deployed multi-node endpoint using Python:

import openai

client = openai.OpenAI(
    base_url="http://<your-eks-alb-ingress-ip>/v1",
    api_key="dummy-key-for-ray-serve"
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    messages=[
        {"role": "user", "content": "Explain the benefits of disaggregated prefill and decode in LLM inference."}
    ],
    temperature=0.7,
    max_tokens=512
)

print(response.choices[0].message.content)

Performance Benchmarking and Optimization

When operating multi-node clusters across Amazon EKS, monitoring network bottlenecks is crucial. Here are several pro tips to ensure maximum throughput:

• EFA Tuning: Ensure that Elastic Fabric Adapter drivers and the AWS OFI NCCL plugin are correctly loaded inside your container to bypass TCP overhead during tensor parallel weight synchronization.
• KV Cache Optimization: Adjust vLLM memory utilization flags (--gpu-memory-utilization 0.90) to prevent out-of-memory errors during peak concurrent request loads.
• Hybrid Routing: If your infrastructure experiences unpredictable traffic spikes, consider implementing a dual-routing strategy that offloads overflow traffic to n1n.ai during peak hours.

Conclusion

Deploying large language models across multi-node Kubernetes clusters no longer requires assembling fragmented open-source libraries from scratch. By combining Amazon EKS, Ray LLM Deep Learning Containers, and disaggregated prefill-decode architectures, engineering teams can achieve enterprise-grade scalability, low latency, and high resource efficiency.

Get a free API key at n1n.ai