Building a Physical AI Model Factory with NVIDIA Cosmos 3 on SageMaker HyperPod

Building autonomous systems, robotics, and embodied AI solutions requires moving beyond traditional static datasets. Physical AI relies on continuous perception-action feedback loops and physical world simulation. Rather than running isolated training jobs, production-grade Physical AI demands a continuous model factory—an automated, self-healing pipeline capable of synthetic data generation (SDG), distributed post-training, and closed-loop evaluation.

In this comprehensive guide, we explore how to construct a resilient Physical AI model factory by pairing NVIDIA Cosmos 3 foundational world models with Amazon SageMaker HyperPod on Amazon EKS. We will break down the end-to-end architecture, dive into code implementations for automated cluster management, and examine strategies to maximize GPU Goodput under heavy physical simulation workloads.

Physical AI Pipeline Architecture

Unlike traditional natural language processing (NLP) or 2D computer vision, Physical AI foundational models must internalize spatial geometry, kinematics, material dynamics, and temporal continuity. NVIDIA Cosmos 3 serves as a foundational world model platform, generating physics-aware video trajectories and predicting future environment states based on robotic actions.

To manage this computationally intensive loop, Amazon SageMaker HyperPod provides persistent, resilient compute infrastructure built directly on Kubernetes (Amazon EKS). The model factory consists of three synchronized stages:

+-----------------------------------------------------------------------------------+
|                         PHYSICAL AI MODEL FACTORY                                 |
+-----------------------------------------------------------------------------------+
|  Stage 1: Synthetic Data Gen (SDG)  | Stage 2: Resilient Post-Training | Stage 3: Closed-Loop Eval    |
|  - NVIDIA Cosmos 3 World Model      | - SageMaker HyperPod on EKS      | - Isaac Sim / Multi-Agent    |
|  - Action-conditioned Trajectories  | - FSDP / Megatron-LM / Goodput   | - Multi-LLM Judge (n1n.ai)   |
+-----------------------------------------------------------------------------------+

Synthetic Data Generation (SDG): NVIDIA Cosmos 3 processes control policies to stream physical trajectories and vision-language-action (VLA) tokens.
Resilient Post-Training: SageMaker HyperPod orchestrates multi-node GPU clusters, automatically recovering from hardware failures while fine-tuning downstream policy heads.
Closed-Loop Evaluation: Simulated environments test policy trajectories. Visual outputs and sensor logs are evaluated by high-throughput LLM judges accessible via unified API gateways like n1n.ai.

Component Breakdown: Cosmos 3 & SageMaker HyperPod

Architecture Layer	Core Technology	Primary Function in Model Factory
World Model Backbone	NVIDIA Cosmos 3	Generates physical video predictions, spatial embeddings, and action-conditioned state transitions.
Orchestration Layer	Amazon SageMaker HyperPod (EKS)	Maintains long-running Kubernetes pods, automates node replacement, and eliminates training downtime.
Data Ingestion & Storage	AWS S3 Express One Zone + FSx for Lustre	Delivers high-bandwidth sub-millisecond data pipelines for video frames and state vectors.
Evaluation & Feedback	Multi-LLM API Gateway (n1n.ai)	Route multimodal trajectories to Claude 3.5 Sonnet / DeepSeek-V3 for reasoning validation.

Step 1: Deploying the SageMaker HyperPod Cluster on EKS

SageMaker HyperPod abstracts node health checks, automatic deep-health diagnostics, and automatic replacement of faulty GPU instances (e.g., p5.48xlarge or g6.12xlarge).

Below is an infrastructure-as-code snippet using the AWS SDK for Python (Boto3) to configure a resilient HyperPod cluster running on EKS:

import boto3

sm_client = boto3.client('sagemaker', region_name='us-west-2')

def create_physical_ai_hyperpod_cluster():
    response = sm_client.create_cluster(
        ClusterName='cosmos-physical-ai-factory',
        InstanceGroups=[
            {
                'InstanceGroupName': 'cosmos-sdg-workers',
                'InstanceType': 'ml.g6.12xlarge',
                'InstanceCount': 8,
                'LifeCycleConfig': {
                    'SourceS3Uri': 's3://my-physical-ai-bucket/scripts/bootstrap-sdg.sh',
                    'OnCreate': 'bootstrap-sdg.sh'
                },
                'ExecutionRole': 'arn:aws:iam::123456789012:role/SageMakerHyperPodRole',
                'ThreadsPerCore': 1
            },
            {
                'InstanceGroupName': 'cosmos-training-nodes',
                'InstanceType': 'ml.p5.48xlarge',
                'InstanceCount': 16,
                'LifeCycleConfig': {
                    'SourceS3Uri': 's3://my-physical-ai-bucket/scripts/bootstrap-training.sh',
                    'OnCreate': 'bootstrap-training.sh'
                },
                'ExecutionRole': 'arn:aws:iam::123456789012:role/SageMakerHyperPodRole'
            }
        ],
        NodeRecovery='Automatic',
        Orchestrator={
            'Eks': {
                'ClusterArn': 'arn:aws:eks:us-west-2:123456789012:cluster/hyperpod-eks-physical-ai'
            }
        }
    )
    return response

if __name__ == '__main__':
    cluster_info = create_physical_ai_hyperpod_cluster()
    print(f"Cluster ARN created: {cluster_info['ClusterArn']}")

Step 2: Continuous Synthetic Data Generation with NVIDIA Cosmos 3

NVIDIA Cosmos 3 enables Physical AI models to predict future video frames given an initial observation frame I_0 and a control action sequence A_{0:T}. In our continuous factory, worker nodes stream action parameters and retrieve predicted environment transformations to build massive multi-modal training sets.

Here is a Python script executing physical trajectory inference using the Cosmos 3 World Model SDK:

import torch
from nvidia_cosmos_sdk import CosmosWorldModel, PipelineConfig

def generate_synthetic_trajectory(initial_frame: torch.Tensor, actions: torch.Tensor):
    # Initialize NVIDIA Cosmos 3 Physics-Aware Tokenizer & Diffusion Backbone
    model = CosmosWorldModel.from_pretrained(
        "nvidia/cosmos-3-wm-7b",
        torch_dtype=torch.bfloat16
    ).to("cuda")
    
    # Configure generation parameters for physical accuracy
    config = PipelineConfig(
        num_inference_steps=30,
        guidance_scale=7.5,
        physics_consistency_weight=0.85
    )
    
    with torch.no_grad():
        # Condition future frames on robotic gripper actions
        predicted_frames = model.generate_trajectory(
            context_frame=initial_frame,
            action_sequence=actions,
            config=config
        )
        
    return predicted_frames

# Example execution within the SDG pipeline pod
if __name__ == '__main__':
    dummy_frame = torch.randn(1, 3, 256, 256, device="cuda", dtype=torch.bfloat16)
    dummy_actions = torch.randn(1, 10, 6, device="cuda", dtype=torch.bfloat16) # 10 steps, 6-DoF action space
    
    frames = generate_synthetic_trajectory(dummy_frame, dummy_actions)
    print(f"Generated frame tensor batch: {frames.shape}")

Step 3: Maximizing GPU Goodput with Resilient Post-Training

In large-scale distributed training on hundreds of GPUs, node crashes and silent data corruption are statistical certainties. GPU Goodput measures the ratio of useful training computation time to total elapsed time:

Optimization Strategies for HyperPod

In-Memory Checkpointing via S3 Express One Zone: Traditional checkpointing to persistent storage halts compute workers. SageMaker HyperPod integrates distributed state save routines directly with S3 Express One Zone, cutting checkpoint save times from 8 minutes down to < 25 seconds for 100B+ parameters.
KubeFlow Operator Integration: Running Kubernetes native jobs on SageMaker HyperPod allows auto-replacement of degraded nodes within 90 seconds without restarting the entire PyTorch Distributed Launcher state.

# Kubernetes Manifest snippet for hyperpod continuous fine-tuning
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: cosmos-physical-vla-finetune
  namespace: hyperpod-training
spec:
  pytorchReplicaSpecs:
    Worker:
      replicas: 16
      template:
        spec:
          containers:
          - name: pytorch
            image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/cosmos-vla:latest
            command: ["python

Source: https://aws.amazon.com/blogs/machine-learning/build-a-physical-ai-model-factory-with-nvidia-cosmos-3-on-sagemaker-hyperpod/