Inference

An analysis of how vLLM's shift toward custom kernels impacts torch.compile compatibility and what developers need to know to maintain performance.

Discover how learned routing and SSD streaming allow massive 35B MoE models to run on consumer hardware with just 3GB of RAM, breaking the memory bottleneck.

A deep dive into the Amazon SageMaker HyperPod Inference Gateway, exploring how its Kubernetes-native routing optimizes GPU utilization and reduces first-token latency for enterprise LLM deployments.

An in-depth analysis of how specialized inference hardware is reshaping LLM API performance and cost efficiency for developers.

A deep dive into integrating Helion support with Hugging Face Kernels to achieve high-performance, portable model inference.

A technical guide on leveraging model caching in Amazon SageMaker HyperPod to reduce inference cold starts by moving model weights to local NVMe storage.

Explore how the partnership between Hugging Face and Cerebras is revolutionizing real-time voice AI by leveraging Gemma models and wafer-scale hardware for sub-100ms latency.

NVIDIA's new Nemotron-Labs Diffusion models revolutionize LLM inference by enabling three generation modes—autoregressive, diffusion, and self-speculative—within a single checkpoint. Achieve up to 6.4x throughput gains without changing your application stack.

South Korean AI chipmaker Rebellions has raised $400 million in a pre-IPO round, valuing the company at $2.3 billion. The startup aims to challenge Nvidia's market dominance by specializing in high-efficiency AI inference chips.