Accelerate agentic AI with NVIDIA Nemotron models on Amazon EKS and SageMaker HyperPod EKS

Deploying state-of-the-art large language models requires a balance of high throughput, low latency, and robust orchestration. As enterprises shift from simple chat interfaces to complex, multi-step agentic workflows, the underlying infrastructure must support advanced reasoning, tool calling, and multimodal perception. The NVIDIA Nemotron 3.5 Super VL—a massive 120-billion-parameter mixture-of-experts model—excels in these exact domains. However, orchestrating such a massive model in production demands resilient cloud-native infrastructure.

In this deep-dive review, we examine the architectural highlights of NVIDIA Nemotron models and demonstrate how to optimize their deployment on Amazon Elastic Kubernetes Service (EKS), EKS Auto Mode, and Amazon SageMaker HyperPod EKS. Furthermore, we will look at how developers can bypass complex infrastructure setups entirely by routing their requests through high-performance LLM API aggregators like n1n.ai for instant development and testing.

Architectural Overview of NVIDIA Nemotron 3.5 Super VL

The NVIDIA Nemotron 3.5 Super VL represents a generational leap in open-weights model design. With 120 billion parameters distributed across a Mixture-of-Experts (MoE) architecture, it routes tokens through specialized expert sub-networks, ensuring high compute efficiency without sacrificing capacity.

Key architectural highlights include:

• Multimodal Integration: Native handling of interleaved text, high-resolution images, and structured tool definitions.
• Advanced Reasoning Engine: Pre-trained and aligned explicitly for step-by-step chain-of-thought (CoT) reasoning, making it ideal for autonomous AI agents.
• Optimized Tokenization: Custom vocabulary tailored for code generation, API schemas, and structured JSON outputs.

When building agentic applications, agents must query multiple tools, parse messy inputs, and maintain context across long execution horizons. Nemotron's large context window and MoE routing make token latency < 50ms per token achievable when paired with the right inference serving engine.

Infrastructure Setup: Amazon EKS and SageMaker HyperPod EKS

Running a 120B parameter model requires distributed inference across multiple GPUs (typically a node pool of NVIDIA H100 or H200 accelerators). AWS provides two primary paths for orchestrating this at scale: Amazon EKS with vLLM, and SageMaker HyperPod EKS for fault-tolerant training and heavy inference scaling.

Deploying with vLLM and NVIDIA Dynamo on EKS

To maximize GPU memory bandwidth and compute utilization, we deploy Nemotron using vLLM combined with NVIDIA Dynamo for dynamic batching and tensor parallelism.

First, ensure your EKS cluster has node groups provisioned with appropriate GPU instances (e.g., p5.48xlarge). Here is a sample Kubernetes deployment manifest configured for tensor parallelism across 8 GPUs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nemotron-vllm-server
  namespace: ai-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nemotron-vllm
  template:
    metadata:
      labels:
        app: nemotron-vllm
    spec:
      containers:
      - name: vllm-container
        image: vllm/vllm-openai:latest
        command: ["python3