Distributed Training

A deep dive into optimizing VRAM usage and scaling LLM training across multiple GPUs using Gradient Accumulation and Distributed Data Parallelism (DDP) in PyTorch.

A deep dive into identifying and resolving data transfer bottlenecks in large-scale AI training using NVIDIA Nsight Systems and modern optimization techniques.

Discover how Mosaic enables 150,000-token sequence processing by sharding attention across multiple GPUs, overcoming the quadratic memory bottleneck.