vLLM vs TGI vs Ollama: GPU Memory Throughput, PagedAttention and Batching Benchmarks

When deploying large language models to production, selecting the right inference engine dictates your infrastructure costs, latency percentiles, and system stability. All three primary serving systems serve LLM inference, but they occupy non-overlapping points on the production spectrum. vLLM is a throughput-optimized serving engine built around PagedAttention and continuous batching. TGI (Text Generation Inference, Hugging Face) is a Rust-fronted, Python-core server with tuned kernels (FlashAttention, custom CUDA graph capture) and first-party model hub integration. Ollama is a llama.cpp-based local runner, optimized for GGUF quantized models, CPU/hybrid execution, and zero-ops, but it is not designed for concurrent serving at all.

Executive Summary & Decision Matrix

Choosing the correct backend requires looking past micro-benchmarks and evaluating operational reality. The decision matrix can be summarized as: Run vLLM if you serve >10 requests/second from a dedicated GPU fleet; run TGI if you need enterprise support, Ranger-level model hub integration, and SAFETENSORS-native weight streaming; run Ollama only for local development, air-gapped single-user inference, or laptop-class prototyping — never for concurrent production traffic.

Ratings: vLLM 4.8/5 (raw serving performance, ecosystem velocity), TGI 4.5/5 (operational maturity, support), Ollama 3.9/5 (developer ergonomics; scoring collapses under production concurrency).

Memory Architecture & PagedAttention Mechanics

Autoregressive decoding stores key/value tensors for every token in the context. With fp16 and a 32-layer, 32-head, 128-dim-head architecture (Llama-3-8B class), the KV cache per token is calculated as:

KV_per_token = 2 (K+V) * num_layers * num_kv_heads * head_dim * dtype_bytes = 2 * 32 * 8 * 128 * 2 = 131 KB/token (GQA, Llama-3-8B) = 2 * 32 * 32 * 128 * 2 = 524 KB/token (MHA, no GQA)

A naive server pre-allocates a contiguous buffer per sequence for max_seq_len. At 8,192 tokens that is 1.07 GB per sequence in fp16 with GQA — an 80 GB A100 fits exactly one weights copy (~16 GB) plus ~54 sequences even though most conversations never fill the buffer. Internal fragmentation routinely wastes 60-80% of the reservation.

vLLM borrows virtual-memory paging from operating systems. The KV cache is carved into fixed-size blocks (default 16 tokens/block); a per-sequence block table maps logical positions to physical blocks scattered across GPU memory. Consequences include:

1. Fragmentation drops below 4% (internal only, bounded by block size). Physical blocks are shared across sequences via copy-on-write when multiple samples fork from the same prefix (beam search, n-parallel sampling).
2. Prefix caching: identical system prompts reuse physical blocks, cutting TTFT up to 8x for long shared prompts.
3. Continuous (iteration-level) batching: the scheduler re-forms the batch every decode step. A finished sequence yields its slot mid-forward-pass; no waiting for the batch to drain. This is where the 3,480 vs 2,910 tok/s gap over TGI largely originates — TGI's scheduler is also continuous since v2, but its token-tree and watermark heuristics are more conservative.

Concurrency and Scaling Benchmarks

When scaling concurrent requests, throughput characteristics diverge sharply between the engines. Developers looking for maximum throughput often integrate n1n.ai to route requests efficiently across multi-backend setups or manage traffic spikes seamlessly.

Notes: vLLM throughput at 128 concurrent is KV-cache-bound (block pool saturated, requests queue in waiting state). Ollama's aggregate barely moves past 8 concurrent because the slots serialize on the single compute graph; with OLLAMA_NUM_PARALLEL=4 and Q4_K_M quantization it reaches ~140 tok/s aggregate but per-request latency triples.

Production Troubleshooting & Pro Tips

When scaling inference in production, engineers frequently encounter subtle bottlenecks that can degrade performance if left unaddressed. Leveraging infrastructure aggregators like n1n.ai can mitigate some of these scaling issues by abstracting away low-level cluster management.

1. vLLM preemption thrash under memory pressure: When the block pool runs dry, vLLM preempts sequences — recomputing KV (or swapping to CPU) for the victim. Under sustained overload with gpu_memory_utilization=0.95 and long contexts, preemption loops collapse throughput by 60-90% while GPU utilization stays at 100%. Debug with the /metrics endpoint (num_preemptions_total) and fix by capping max_num_seqs and lowering gpu_memory_utilization to 0.90.
2. TGI batch watermark + OOM cascades: TGI's memory watermark assumes text-model activation shapes; loading a MoE or a vision-language model with --max-batch-size tuned for a dense model produces CUDA OOM at traffic levels the watermark said were safe. Mitigation: compute --max-batch-total-tokens from actual activation profiles.
3. Ollama model unloading silently resets context: Default keep_alive is 5 minutes; after idle, weights are evicted and the next request pays a full reload. Set keep_alive=-1 on any shared host and pin OLLAMA_NUM_PARALLEL explicitly.

Conclusion

The three tools are not substitutes. vLLM remains the default production engine with a 4.8/5 score, offering the highest tokens/s per GPU dollar. TGI scores 4.5/5 as the operationally robust choice for enterprise deployments. Ollama scores 3.9/5, perfect for local developer machines but architecturally outmatched under concurrent production traffic.

Get a free API key at n1n.ai