Benchmarking Our Custom LLM Inference Engine Against vLLM and SGLang

When running demanding models like GLM-5.3 under sustained load on 8× B200 GPUs with 32 concurrent workflows, standard inference engines can hit severe bottlenecks. In our tests, the median request waiting time for the first token reached an astonishing 357 seconds on vLLM. By comparison, our custom inference engine running the exact same model on the same hardware hardware achieved a median Time to First Token (TTFT) of just 3.24 seconds.

This post details how we built our own inference engine from scratch—not as a vLLM fork, and not relying on quantized serving models—to compete directly against vLLM and SGLang across four models and three hardware configurations.

Architecture and Design Philosophy

Most enterprise teams scaling large language models face a hard physical limit: they cannot easily acquire more GPUs. Consequently, maximizing hardware utilization and squeezing every drop of performance out of existing infrastructure is paramount. We designed our engine with two non-negotiable rules:

1. No Precision Shortcuts: We run full model weights (BF16) rather than resorting to quantized serving models (like FP8) to fake speed gains.
2. Full Weight Verification: Speculative decoding uses a proprietary 4-bit draft head, but the base BF16 model meticulously checks every single token drafted to guarantee uncompromised output quality.

To demonstrate the real-world impact, let us examine how our engine fares against established open-source alternatives across different models and hardware setups.

Benchmark Results Across Models

1. GLM-5.3 on 8× B200 GPUs

Under a rigorous 30-minute window targeting 32 active workflows simultaneously, both engines processed the initial batch of 116 workflows. However, the divergence in performance was stark:

Because our engine cleared tasks significantly faster, it continued ingesting new workflows seamlessly, whereas vLLM stalled after processing the initial queue. During burst tests dropping 96 workflows concurrently, our engine completed 95 out of 96 requests, delivering a 2.61× throughput advantage and a 10× improvement in median TTFT.

2. Qwen3.6-35B-A3B on RTX PRO 6000 Blackwell

Tested on a single Google Cloud instance at BF16 precision with 3 MTP draft tokens and 32K context length, we evaluated concurrency scaling from 8 to 256 requests against a heavily tuned vLLM configuration locked at 95% GPU memory utilization:

Across all concurrency tiers, our throughput exceeded vLLM by over 50%, with median TTFT reaching up to 5.75× faster under steady traffic and 8.56× faster during traffic bursts.

3. Qwen3.8-27B vs vLLM and SGLang

With a smaller model headroom, performance differentials tend to compress. To maintain rigorous objectivity, we expanded our test matrix to include SGLang, running 8 workload points across 13 distinct metrics (104 total comparisons per engine):

• vs vLLM 0.33: Won 104 out of 104 metrics.
• vs SGLang: Won 102 out of 104 metrics.

Overall throughput surpassed vLLM by up to 44% and SGLang by up to 26%, with every mean, p50, p90, and p99 latency metric favoring our engine. For instance, at 1K input tokens and 64 concurrent requests, our median TTFT plummeted to 299 ms compared to vLLM's 13.3 seconds.

4. Agentic Workloads with Qwen3.8-Flash-Next on 8× H100

Standard chat benchmarks fail to capture modern production workflows involving coding agents, multi-turn dialogue, shared context, and subagents. Using the SemiAnalysis AgentX simulation framework—where average inputs ranged from 87K to 106K tokens per request—we observed massive gains:

• p90 Interactivity (1 session): 209.0 vs 16.3 tok/s per user (12.8× improvement).
• p90 Interactivity (12-16 sessions): Maintained a 5.0× to 7.0× advantage.
• p90 TTFT (1 session): 0.78 seconds compared to 10.6 seconds.

Remarkably, our engine achieved these results running at full BF16 precision, whereas the SGLang reference utilized FP8 quantization. Even with their quantization shortcut, our architecture prevailed.

Integrating High-Performance Inference in Production

Building custom inference engines on clusters of B200 or H100 GPUs is an extraordinary engineering feat, but production applications need seamless API access without managing underlying infrastructure headaches. Whether you are orchestrating multi-agent systems using frameworks like LangChain or executing complex RAG pipelines, having reliable access to cutting-edge models like DeepSeek-V3 or Claude 3.5 Sonnet is vital.

Platforms like n1n.ai aggregate top-tier LLM endpoints to deliver the low latency and high availability enterprise applications demand. When developing enterprise applications with n1n.ai, developers can easily switch between model providers and benchmark speeds programmatically.

Here is a quick Python snippet demonstrating how to interact with an optimized LLM endpoint via the standard OpenAI-compatible SDK supported by aggregators like n1n.ai:

from openai import OpenAI

# Initialize client with n1n.ai aggregator endpoint
client = OpenAI(
    base_url="https://api.n1n.ai/v1