Training a 1T-Parameter MoE on EU Public Infrastructure with PyTorch and TorchTitan

The landscape of Large Language Model (LLM) pretraining has long been dominated by tech giants with proprietary, tightly coupled hardware clusters. However, a significant milestone was recently achieved by a consortium of European researchers and the PyTorch team: the successful pretraining of a 1.1-trillion-parameter Mixture-of-Experts (MoE) model on shared public infrastructure. This feat, utilizing the Leonardo supercomputer in Italy, demonstrates that the combination of PyTorch and the emerging TorchTitan library can overcome the traditional bottlenecks of high-latency interconnects and shared filesystem constraints.

The Shift to Mixture-of-Experts (MoE) at Scale

As models scale toward the trillion-parameter mark, the computational cost of standard dense Transformers becomes prohibitive. Mixture-of-Experts (MoE) architectures, such as those powering DeepSeek-V3 or Mixtral, offer a solution by activating only a fraction of the total parameters for any given input token. While this improves inference efficiency, it introduces massive complexity in training. Specifically, MoE models require "Expert Parallelism" (EP), where different experts are hosted on different GPUs, necessitating high-bandwidth "All-to-All" communication patterns.

For developers looking to integrate these massive MoE capabilities into their applications without the astronomical cost of training, n1n.ai offers a streamlined gateway to the world's most powerful LLM APIs, ensuring that even trillion-parameter logic is accessible via a single integration point.

Enter TorchTitan: Modular Scaling for the Modern Era

Historically, training models of this magnitude required monolithic frameworks like Megatron-LM. While powerful, these frameworks are often difficult to customize. TorchTitan represents a paradigm shift. It is a clean-slate, modular repository built on top of PyTorch's native distributed primitives (FSDP2, Tensor Parallel, Pipeline Parallel).

TorchTitan allows for "4D Parallelism," which was critical for the 1.1T MoE run. The 4D strategy includes:

1. Data Parallelism (FSDP2): Sharding model states across data-parallel ranks to save memory.
2. Tensor Parallelism (TP): Splitting individual weight matrices across multiple GPUs to handle massive hidden dimensions.
3. Pipeline Parallelism (PP): Dividing model layers across different stages to reduce the memory footprint per device.
4. Expert Parallelism (EP): Specifically for MoE, ensuring that experts are distributed efficiently to minimize communication overhead.

Overcoming Infrastructure Constraints on Leonardo

The Leonardo supercomputer, part of the EuroHPC initiative, presents a different environment than a dedicated corporate cluster. It uses the Atos BullSequana XH2000 architecture with NVIDIA A100 GPUs. The primary challenge on such shared infrastructure is the interconnect. Unlike the ultra-low-latency InfiniBand setups in private labs, shared public clusters often face higher contention and variable latency.

To achieve a stable Model Flops Utilization (MFU), the team utilized PyTorch's DeviceMesh to create a hierarchical topology. By nesting TP within a single node and applying EP and FSDP across nodes, they minimized the impact of cross-node communication bottlenecks. This level of fine-grained control is exactly why platforms like n1n.ai are becoming essential for the ecosystem; they abstract the underlying infrastructure complexity while providing high-performance access to the resulting models.

Technical Implementation: Memory and Communication Optimization

Training a 1.1T model with {num_experts: 64} and a hidden size of < 8192 requires meticulous memory management. The team implemented several key optimizations in TorchTitan:

• Selective Activation Checkpointing: Instead of storing all intermediate activations for the backward pass, only the most memory-intensive layers were recomputed. This reduced memory usage by nearly 40% with a minimal hit to throughput.
• FP8 Training: Leveraging the hardware capabilities of modern GPUs, the team integrated FP8 precision for specific kernels, significantly speeding up the GEMM operations in the MoE routers.
• Asynchronous All-to-All: By overlapping the expert routing communication with the computation of the experts themselves, the researchers hid a large portion of the network latency.

Benchmarking Performance

The results were impressive. On 128 nodes (512 A100 GPUs), the 1.1T MoE model achieved a throughput that rivaled dense models of much smaller size. The Model Flops Utilization (MFU) remained stable above 45%, a high figure for shared public infrastructure. This proves that with the right software stack—specifically PyTorch 2.4+ and TorchTitan—the barrier to entry for "Sovereign AI" is lower than ever.

For enterprises that need the reasoning power of a 1T-parameter model but lack the 512-GPU cluster, n1n.ai provides the necessary infrastructure to query models like Claude 3.5 Sonnet or OpenAI o3, which share similar architectural complexities but are delivered as a managed service.

Pro Tip: Configuring TorchTitan for MoE

To replicate these results on a smaller scale, developers can use the following TorchTitan configuration snippet:

# Example TorchTitan config for MoE
model:
  name: "moe_model"
  num_layers: 32
  hidden_size: 4096
  num_experts: 8
  top_k: 2

parallelism:
  tensor_parallel_size: 2
  pipeline_parallel_size: 2
  expert_parallel_size: 4
  enable_fsdp: True

Conclusion

The success of the 1.1T MoE training on EU public infrastructure is a testament to the maturity of the PyTorch distributed ecosystem. By moving away from monolithic frameworks and toward modular libraries like TorchTitan, the AI community is democratizing the ability to train at the absolute limit of scale. Whether you are training your own 1T model or leveraging the best-in-class models via n1n.ai, the tools for the next generation of AI are now in the hands of the many, not just the few.

Get a free API key at n1n.ai