StepFun Step 5 Preview: Open Weights 600B MoE with 27B Active Parameters and 1M Token Context

The engineering of open weights models has reached a new maturity milestone. StepFun has released the open weights for Step 5 Preview, its most ambitious foundational model tailored for long-horizon agentic orchestration and multi-hop analytical problem-solving. Distributed in BF16, FP8, and NVFP4 quantization, Step 5 establishes a new benchmark among self-hostable open models for enterprise deployment.

Featuring 600 billion total parameters with a sparse activation of only 27 billion per token, Step 5 Preview adopts a deliberately narrow and deep topology: 92 layers interspersing Gated Sliding Window Attention (Gated SWA) and Gated Grouped-Query Attention (Gated GQA). This architecture enables a context window of up to 1 million tokens without prohibitive VRAM explosion, drastically cutting prefill costs in coding and tool-calling audit workflows.

When scaling LLM workloads in enterprise environments, developers often rely on reliable intermediaries. Utilizing an LLM API aggregator like n1n.ai ensures maximum throughput, lowest latency, and unified access to models across providers. Get a free API key at n1n.ai.

Architectural Breakdown: Narrow, Deep, and Sparse

Unlike conventional models that widen hidden dimensions to pack more representation capacity, Step 5 Preview invests heavily in layer depth. In complex agentic tasks, the primary bottleneck isn't immediate associative memory, but the ability to sustain chained deductive reasoning across multiple consecutive tool calls and compiler outputs.

To balance the prefill overhead that a 92-layer stack imposes on extended contexts, StepFun adopted a 3:1 interleaving ratio between attention mechanisms:

1. Gated Sliding Window Attention (Gated SWA — 3 out of 4 layers): Instead of calculating full cross-attention across all token pairs in a 1-million-token context, Gated SWA restricts quadratic computation to a fixed sliding horizon, complemented by linear multiplicative gates.
2. Gated Grouped-Query Attention (Gated GQA — 1 out of 4 layers): Every three sliding window layers, Step 5 injects a full attention layer with grouped keys and values, connecting long-distance dependencies across the entire document.

Sparse routing operates via a classifier selecting 27 billion useful parameters per emitted token, combining shared experts (universal syntactic representations) and specific experts (abstract reasoning).

Frontier Benchmark Performance

To gauge the real-world competitiveness of Step 5 Preview against dominant international models, the community subjected the open weights to standardized evaluations. The comparison pitted it against contemporaries such as DeepSeek 4.1, OpenAI GPT-6 Astra, Claude Fable 5.1, and Gemini 3.8 Flash Cyber.

The numbers highlight strategic advantages:

• Software Engineering Parity: SWE-bench Verified at 78.2%, slightly outperforming GPT-6 Astra (78.1%).
• Symbolic Math Accuracy: AIME 2026 at 94.1%, benefiting from an agent-focused post-training curriculum.
• Agentic Robustness: AgentBench at 87.6%, showing exceptional resilience in long tool-calling loops.

When routing production API traffic across multiple open and closed models, integrating an enterprise aggregator like n1n.ai simplifies fallback mechanisms and cost tracking.

Python Asynchronous Client Implementation

Below is a non-blocking Python client compatible with vLLM or SGLang servers hosting Step 5 Preview, supporting Pydantic schemas and Model Context Protocol (MCP 2.0) tool definitions.

import os
import asyncio
import logging
from typing import Dict, Any, List, Optional
import httpx
from pydantic import BaseModel, Field

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("step5_client")

class MCPToolDefinition(BaseModel):
    name: str = Field(..., description="Tool identifier in MCP 2.0")
    description: str = Field(..., description="Functional description for agentic invocation")
    parameters: Dict[str, Any] = Field(..., description="JSON Schema parameters")

class Step5AsyncClient:
    def __init__(self, base_url: Optional[str] = None, api_key: Optional[str] = None):
        self.base_url = base_url or os.getenv("STEP5_ENDPOINT", "http://localhost:8000/v1")
        self.api_key = api_key or os.getenv("STEP5_API_KEY", "EMPTY")
        self.headers = {"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"}

    async def generate(self, prompt: str) -> str:
        url = f"{self.base_url}/chat/completions"
        payload = {
            "model": "stepfun-ai/Step-5-Preview",
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.2,
            "max_tokens": 4096
        }
        async with httpx.AsyncClient(timeout=180.0) as client:
            response = await client.post(url, headers=self.headers, json=payload)
            response.raise_for_status()
            return response.json()["choices"][0]["message"]["content"]

if __name__ == "__main__":
    client = Step5AsyncClient()
    # asyncio.run(client.generate("Analyze distributed sorting algorithm."))
}

Enterprise Hosting and Deployment Strategy

Self-hosting a 600B parameter model requires rigorous hardware planning. StepFun collaborated with the open-source community to ensure day-0 support across major serving engines:

• vLLM with PagedAttention: Tensor parallel size 8 (TP=8) enables single-node serving with low latency for up to 128k token prompts.
• SGLang with RadixAttention: KV cache tree management reduces Time-to-First-Token by up to 53% in agentic search trees.
• NVFP4 Quantization via TensorRT-LLM: Reduces VRAM footprint to ~340 GB, allowing deployment on 4 Hopper or Blackwell accelerators.

Whether you are self-hosting open models or routing requests through cloud endpoints, scaling infrastructure efficiently is critical. For a seamless developer experience combining self-hosted checkpoints and managed LLM endpoints, leverage n1n.ai.

Get a free API key at n1n.ai