How to Run a 110B LLM on 16GB RAM: The Math Behind Local Inference Speed

Running a 110-billion parameter model like GLM-4.5-Air on a 2016 desktop with only 16GB of RAM sounds like a fever dream. Yet, by understanding the underlying physics of memory bandwidth and the mathematical fragility of model layers, it is not only possible but predictable. This tutorial explores the 'Tiered Decode Law' and the tools you need to optimize local LLMs, while explaining when you should transition to high-performance cloud solutions like n1n.ai.

The Experiment: Placement Over Budget

Most developers assume that to make a model smaller, you simply need a better quantizer (like GPTQ or AWQ). However, experimental data suggests that where you spend your bits matters more than how many bits you spend.

In a control experiment using two Q2_K GGUF files of Gemma 4 12B, both identical in size (5.22 GB), we see a massive discrepancy in quality (measured by WikiText-2 Perplexity):

By simply moving the protected (higher-precision) blocks to the end of the stack, we achieved a -2.25 PPL swing. This improvement is equivalent to doubling the byte budget, proving that the "spend more bits everywhere" instinct is inefficient. For developers building production-grade RAG pipelines, these local optimizations are great for testing, but for scaling, the API reliability of n1n.ai remains the gold standard.

The Four Laws of Local LLM Inference

Through months of falsification experiments on a legacy i5-7600K system with a GTX 1060, four fundamental laws have emerged:

1. Rotation is Rank-Conditional: Incoherence rotation (like QuIP#) is not a universal good. It can cost +0.006 PPL on full-rank MLPs but a catastrophic +1623 PPL on low-rank KV-latents.
2. Trained Networks are Dense Everywhere: There is no "free" sparsity. Routing in Mixture-of-Experts (MoE) models like DeepSeek-V3 is domain-flat; prose and code use nearly identical expert sets. 2-bit quantization is the absolute data-free floor.
3. Fragility is Measurable, Not Predictable: Every model family has a "fragile band." Gemma is late-fragile, while Mistral is early-fragile. You cannot guess this; you must probe it functionally.
4. The Tiered Decode Law: This is the master equation for speed: tok/s = η(tier) × bandwidth ÷ active-bytes-per-token

In this equation, η (eta) represents the efficiency coefficient of the memory tier (VRAM ≈ 0.56, RAM ≈ 0.29–0.68, Disk ≈ 0.88–1.0). This equation predicted a speed of 0.19 tok/s for a 110B model streaming from a SATA SSD, matching the measured result perfectly.

Implementation Guide: Using quantprobe

To automate these findings, the quantprobe tool allows you to analyze and quantize models based on their specific fragility.

Step 1: Installation

pip install git+https://github.com/FedericoTs/quantprobe

Step 2: Predicting Performance

You can predict your inference speed before even downloading a model. This is crucial for choosing between local deployment and an LLM API aggregator like n1n.ai.

quantprobe plan --model qwen3-30b --machine 2016-xmp
# Output: Predicts ~18.9 tok/s

Step 3: Depth-Aware Quantization

Instead of a uniform 2-bit quantization, we use a regex to protect the fragile layers identified by the probe. For Gemma, the command looks like this:

llama-quantize \
  --tensor-type "blk\.([0-9]|[12][0-9]|3[0-5])\.ffn_.*=q2_k" \
  --tensor-type "blk\.(3[6-9]|4[0-7])\.ffn_.*=q4_k" \
  --tensor-type "attn_.*=q4_k" --token-embedding-type q4_k \
  gemma-4-12B-f16.gguf out-depthaware.gguf Q2_K 8

Pro Tip: Hybrid Placement for MoE

When running MoE (Mixture of Experts) models like DeepSeek-V3 or Qwen3-30B, the placement of experts is vital. On older Pascal-series GPUs, serving experts from CPU RAM instead of VRAM can actually increase performance by 54% due to how llama.cpp handles memory overhead.

However, if your application requires sub-100ms latency for Claude 3.5 Sonnet or OpenAI o3 models, local hardware will always be the bottleneck. In these cases, offloading to n1n.ai provides the necessary throughput without the hardware headache.

Validating the Law

The Tiered Decode Law has been validated from 7B to 744B parameter models. For instance, the Laguna S 2.1 (117.6B MoE) was predicted to run at 47 tok/s on a DGX Spark based purely on its config. The published benchmark was 47.5 tok/s—a 1% variance.

Conclusion

Local LLM optimization is a game of physics and placement. By identifying the fragile layers of a model and understanding your hardware's memory bandwidth tiers, you can run massive models on consumer hardware.

For developers who need to bridge the gap between local experimentation and enterprise-scale deployment, using a robust API aggregator is essential.

Get a free API key at n1n.ai