llama.cpp

An in-depth technical analysis of building a zero-dependency, plug-and-play offline AI USB drive containing seven LLMs. Explores OS runtime bugs, filesystem edge cases, llamafile fork() issues, whisper.cpp audio buffer bugs, and when to pivot to cloud API aggregators like n1n.ai.

Learn how VIDRAFT's POCKET-Darwin-180B model uses graft quantization and SSD offloading in llama.cpp to run a top-ranking 180-billion-parameter MoE model on consumer laptops and mini PCs.

Discover how expert usage sorting, dynamic tier-based quantization, and llama.cpp CUDA kernel patching enable running the 125B Qwen3.8-Flash-Next MoE model on three consumer RTX 3090 GPUs at 80 tokens/sec.

A deep dive into how Transformers now supports GGUF/llama.cpp quantization, enabling efficient local inference for high-performance LLMs.

A comprehensive guide to deploying Qwen 3.8 27B locally, covering GGUF quantization sizes, the hybrid attention KV cache trick, and troubleshooting chat templates for optimal performance.

An in-depth technical deep dive into Meta's Muse Glimmer 30B model, exploring its dense architecture, DFlash speculative decoding, and how it stacks up against Qwen and Gemma in agentic benchmarks.

The official launch of llama.app marks a major milestone for local LLMs, bringing a zero-config web interface to the world's most popular inference engine.

Discover the 'Tiered Decode Law' that predicts LLM inference speed across hardware. Learn how depth-aware quantization allows a 110B model to run on a 2016 desktop with 16GB RAM.

A deep dive into the technical and economic realities of running 4x RTX 3090s for local LLM inference versus utilizing high-performance LLM APIs.

Learn how to deploy Zhipu AI's GLM-5.2 locally using llama.cpp, Ollama, and LM Studio. This guide covers hardware requirements, 2-bit quantization, and why open-weights models are the ultimate insurance against cloud API bans.

Explore the architecture of KathaGPT, a privacy-first local AI desktop application built with Rust and Tauri, featuring a unified inference engine for local and cloud LLMs.

A comprehensive guide on how to utilize the Vulkan backend to run FLUX.1 Schnell and modern LLMs on legacy AMD RX 580 hardware, bypassing the lack of ROCm support.