Deploying GLM-5.2-FP8 (700B MoE) on Modal with 8x H200 GPUs

The release of GLM-5.2 by Zhipu AI marks a watershed moment for the open-weights ecosystem. As a Mixture-of-Experts (MoE) reasoning model specifically engineered for long-horizon planning and complex software engineering, it has quickly ascended the benchmarks. According to recent evaluations like SWE-bench Pro and GPQA, GLM-5.2 is currently the most capable open-source LLM, rivaling or even surpassing proprietary giants like Claude 3.5 Sonnet and GPT-4o in engineering-centric tasks.

However, the sheer scale of this model—a massive 703.74 GiB FP8 checkpoint—presents a significant infrastructure challenge. To serve this model with its full 131k token context window, one must orchestrate an 8x NVIDIA H200 GPU cluster, where each GPU provides 141GB of HBM3e memory. While n1n.ai provides instant access to high-performance LLM APIs for those who want to skip the infrastructure headache, many enterprises require self-hosted solutions for privacy and custom logic. This guide documents the serverless deployment architecture on Modal using vLLM, the technical bottlenecks encountered, and the practical lessons learned during the integration.

The Economics of Serverless H200 Clusters

When dealing with 8x H200 nodes, the costs can escalate quickly. Renting a dedicated node on traditional clouds like RunPod costs roughly $35.12 per hour. In contrast, Modal’s serverless approach charges$36.31 per hour ($0.001261/GPU/sec) but offers a critical advantage: it scales to zero.

In a typical 20-minute active development session, including cold starts and idle wait times, the cost on Modal is approximately $12.00. Once the session ends, the cost drops to exactly$0.00/hour. This makes it an ideal environment for R&D and specialized agentic workflows that don't require 24/7 uptime. For developers who need high-speed inference without managing these clusters, n1n.ai remains the premier choice for stable API access.

Architectural Trade-off Analysis: Quantization Formats

Deploying a 700B parameter model on a single 8-GPU node requires surgical precision in memory layout. Serving the original BF16 weights is mathematically impossible on a single node, as it would require over 1.5 Terabytes of VRAM. We must choose a quantization format that balances accuracy and hardware efficiency.

FP8 is the clear winner here. It leverages NVIDIA Hopper’s native hardware Tensor Cores and utilizes DeepSeek's open-source DeepGEMM library (integrated within vLLM) to execute MoE routing kernels with highly optimized Triton paths. This ensures that the model retains 99.2% of its raw intelligence while fitting comfortably within the 1.1 TB of aggregate VRAM provided by an 8x H200 cluster.

Why Self-Host? The Technical Necessity

While managed providers like n1n.ai offer low friction, certain scenarios mandate self-hosting:

1. Strict Codebase Privacy: Building PoCs in regulated industries (finance/healthcare) often prohibits sending proprietary code to third-party routers.
2. Bypassing Rate Limits: Autonomous agents performing SWE-bench runs require massive context evaluation. Self-hosting ensures the entire 8x H200 compute power is dedicated to your task.
3. Prefix Caching Stability: In a self-hosted environment, you control the RadixAttention prefix cache. Your context stays warm, unlike multi-tenant APIs where caches are constantly evicted to balance load.

Infrastructure-as-Code (IaC) with Modal

To serve GLM-5.2 serverless, we use a specialized vLLM build. Below is the configuration for Modal, emphasizing memory utilization and startup efficiency.

import os
import modal

vllm_image = (
    modal.Image.from_registry(
        "vllm/vllm-openai:glm52-cu129",
        setup_dockerfile_commands=[
            "RUN ln -sf $(which python3) /usr/local/bin/python",
            "RUN rm -f /usr/local/lib/python3.12/dist-packages/typing_extensions.py",
        ],
    )
    .entrypoint([])
    .pip_install("aiohttp", "typing-extensions>=4.15.0")
    .env({"HF_XET_HIGH_PERFORMANCE": "1", "VLLM_LOG_STATS_INTERVAL": "1"})
)

app = modal.App("glm5-2-inference")

@app.function(
    image=vllm_image,
    gpu="H200:8",
    max_replicas=1,
    scaledown_window=15 * 60,
    secrets=[
        modal.Secret.from_name("huggingface"),
        modal.Secret.from_name("vllm-api-key"),
    ],
    volumes={
        "/root/.cache/huggingface": modal.Volume.from_name("huggingface-cache", create_if_missing=True),
    },
)
@modal.web_server(port=8000, startup_timeout=60 * 60)
def serve():
    import subprocess
    cmd = [
        "vllm", "serve", "zai-org/GLM-5.2-FP8",
        "--served-model-name", "glm-5.2-fp8",
        "--host", "0.0.0.0",
        "--port", "8000",
        "--tensor-parallel-size", "8",
        "--kv-cache-dtype", "fp8",
        "--max-model-len", "131072",
        "--gpu-memory-utilization", "0.92",
        "--trust-remote-code",
        "--speculative-config", '\{"method": "mtp", "num_speculative_tokens": 5\}',
        "--safetensors-load-strategy", "prefetch",
        "--enable-prefix-caching",
        "--enforce-eager"
    ]
    subprocess.Popen(cmd)

Key Lessons and Bottlenecks

1. The typing_extensions Conflict

During the initial container boot, we encountered an ImportError. The base CUDA image shipped with a legacy typing_extensions.py file that shadowed our modern package. The resolution required explicitly deleting the legacy file in the Dockerfile setup to allow pydantic-core to find the required Sentinel class.

2. Optimizing Cold Starts

Initially, model loading took over 12 minutes due to sequential reads over the virtual network filesystem. By enabling --safetensors-load-strategy prefetch, we forced vLLM to parallelize the disk-to-VRAM loading process. This reduced model loading time from 12 minutes to ~1 minute, bringing the total cold start (including hardware allocation) to roughly 4.5 minutes.

3. Eager Mode vs. CUDA Graphs

GLM-5.2 uses Multi-Token Prediction (MTP) to speculate 5 tokens ahead. Compiling CUDA graphs for a 131k context window on a 700B model would take over 20 minutes on every startup. We chose --enforce-eager mode. While this causes a ~35-second Time-To-First-Token (TTFT) spike on the very first query while Triton kernels compile, it avoids the massive startup hang, allowing for a more responsive serverless lifecycle.

Performance Validation: The "Sunset Flier" Test

To stress-test the model, we integrated it into OpenCode and tasked it with creating a functional game in a single pass. The prompt required a Flappy Bird clone using only HTML5, CSS, and vanilla JS, with a specific constraint: no external assets.

GLM-5.2 successfully generated "Sunset Flier," utilizing the Web Audio API to synthesize retro sound effects (jumping, scoring, crashing) using oscillator nodes. The logic included gravity acceleration, jump impulses, and high-score persistence. This demonstrated the model's ability to handle complex, multi-modal engineering logic within a single context window.

Future Optimization Vectors

To further refine this deployment, we are looking at three areas:

• Keep-Warm Scheduling: Using a serverless cron job to ping the /health endpoint every 14 minutes to eliminate cold starts during peak hours.
• GPU Memory Snapshots: Modal's snapshotting technology could allow us to save the post-warmup VRAM state to disk, potentially reducing cold starts to under 10 seconds.
• SGLang Migration: Moving from vLLM to SGLang once it natively supports GLM-5.2's MoE layers to reduce CPU overhead during Eager execution.

Self-hosting a 700B reasoning model on serverless infrastructure is no longer a fantasy. With the right orchestration and quantization, developers can access frontier-level intelligence with total data sovereignty.

Get a free API key at n1n.ai