Accelerate Agentic AI with NVIDIA Nemotron Models on Amazon EKS and SageMaker HyperPod EKS

The landscape of artificial intelligence is rapidly shifting from simple zero-shot completions to complex agentic workflows. Enterprises are no longer satisfied with models that merely generate text; they demand autonomous systems capable of reasoning, calling external APIs, executing code, and processing multimodal inputs in real time. To meet these rigorous demands, NVIDIA introduced the Nemotron 3.5 Super VL model, a massive 120-parameter mixture-of-experts architecture specifically engineered for advanced reasoning, robust tool use, and deep multimodal understanding. However, deploying a model of this magnitude in production presents significant infrastructure challenges, including high latency, memory bandwidth bottlenecks, and complex cluster management.

This comprehensive guide explores the architecture of NVIDIA Nemotron models, analyzes their capabilities in agentic workflows, and provides a step-by-step deployment blueprint using Amazon EKS, EKS Auto Mode, and Amazon SageMaker HyperPod EKS, supercharged by NVIDIA Dynamo and vLLM. As developers scale these workloads, managing multiple underlying LLM APIs efficiently becomes critical. Platforms like n1n.ai provide the necessary high-speed infrastructure abstraction to simplify multi-model routing and API orchestration.

Understanding the Architecture of NVIDIA Nemotron 3.5 Super VL

NVIDIA Nemotron 3.5 Super VL stands out in the current generation of open-weights foundation models due to its Mixture-of-Experts (MoE) design. Unlike dense models that activate all parameters for every single token, an MoE architecture selectively routes tokens through a subset of expert networks, drastically reducing computational overhead during inference while maintaining a massive parameter count for deep semantic representation.

The core characteristics of Nemotron 3.5 Super VL include:

• 120 Billion Total Parameters: Delivering high-capacity reasoning capabilities comparable to top-tier proprietary models.
• Multimodal Native Processing: Seamlessly integrating visual and textual inputs without secondary embedding pipelines.
• Advanced Agentic Alignment: Fine-tuned specifically for multi-step reasoning, JSON formatting for tool execution, and code synthesis.

When building agentic AI loops, the model must frequently evaluate intermediate steps, invoke external functions, and parse structured feedback. This iterative process amplifies latency concerns, making high-throughput inference engines a mandatory prerequisite for production deployment.

Infrastructure Prerequisites for Large-Scale MoE Deployment

Deploying a 120B-parameter MoE model requires careful hardware and software orchestration. Typically, this demands multi-node GPU clusters equipped with NVIDIA H100 or A100 Tensor Core GPUs, interconnected via high-bandwidth fabrics like NVLink and InfiniBand.

On the cloud infrastructure side, Amazon Elastic Kubernetes Service (EKS) and Amazon SageMaker HyperPod provide the ideal foundation. EKS offers robust container orchestration with native auto-scaling, while SageMaker HyperPod is specifically tailored for distributed deep learning, offering automated node resilience and fault tolerance for massive training and inference jobs.

To maximize throughput and minimize time-to-first-token (TTFT), developers rely on inference optimization frameworks such as vLLM and NVIDIA Dynamo. vLLM introduces PagedAttention, which virtually eliminates memory fragmentation in the Key-Value (KV) cache, while NVIDIA Dynamo optimizes tensor parallelism across multiple GPU nodes.

Step-by-Step Deployment Guide on Amazon EKS

Deploying Nemotron 3.5 Super VL on Amazon EKS requires setting up the cluster with the NVIDIA GPU Operator, configuring vLLM as the serving backend, and exposing the endpoint via Kubernetes services.

Step 1: Setting up the EKS Cluster with GPU Support

First, ensure your EKS cluster is provisioned with GPU-enabled node groups (e.g., p5.48xlarge instances featuring NVIDIA H100 GPUs). Install the NVIDIA device plugin and GPU Operator:

helm repo add nvidia https://nvidia.github.io/k8s-device-plugin
helm repo update
helm install nv-device-plugin nvidia/nv-device-plugin --namespace kube-system

Step 2: Configuring vLLM Deployment Manifest

Create a Kubernetes deployment manifest (nemotron-deployment.yaml) to run the model server using vLLM. Notice how we configure tensor parallelism to distribute the 120B parameters across multiple GPUs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nemotron-vllm-server
  namespace: ai-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nemotron-vllm
  template:
    metadata:
      labels:
        app: nemotron-vllm
    spec:
      containers:
      - name: vllm-container
        image: vllm/vllm-openai:latest
        command: ["python3