Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Abstract
Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a highly efficient tensor parallelism strategy for transformers that enabled the practical training of multi-billion parameter language models, fundamentally reshaping the ML systems landscape and serving as the foundational blueprint for modern large-scale model training.
Methodology
The paper introduces a highly practical intra-layer model parallelism (tensor parallelism) strategy specifically engineered for Transformer architectures. By partitioning weight matrices and attention computations across GPUs and synchronizing activations via collective communications (primarily all-reduce), the method bypasses the memory ceilings of data parallelism and the pipeline inefficiencies of inter-layer parallelism. The core methodological innovation lies in its simplicity: it requires only minimal PyTorch modifications and standard NCCL collectives, avoiding the need for custom compilers or complex graph transformations. Additionally, the paper provides a critical architectural insight regarding layer normalization placement in BERT-like models, demonstrating that careful positioning is essential for training stability and performance as parameter counts scale into the billions.
Results & evaluation
The experimental design is rigorous and directly addresses the scaling challenges of large transformers. The authors successfully train models up to 8.3 billion parameters across 512 GPUs, reporting detailed systems metrics including sustained throughput (15.1 PFLOPs) and strong scaling efficiency (76% relative to a highly optimized single-GPU baseline). Performance evaluations on WikiText-103, LAMBADA, and RACE establish clear state-of-the-art results, empirically validating that increased model capacity translates to measurable gains when training is stabilized. The ablation studies on layer normalization placement and scaling curves provide actionable empirical guidance for practitioners.
Reproducibility
High. The methodology relies exclusively on native PyTorch operations and standard distributed communication primitives, making it inherently reproducible across standard GPU clusters. The authors subsequently open-sourced the implementation, which became a foundational codebase for the community. The paper provides sufficient architectural specifications, hyperparameter ranges, and communication patterns to replicate the scaling behavior and convergence properties without proprietary tooling.
Limitations
The work primarily focuses on pure intra-layer parallelism and does not extensively explore hybrid strategies (e.g., combining tensor, pipeline, and data parallelism), which later proved necessary for models exceeding 50B parameters. Communication overhead, while minimized, remains sensitive to interconnect topology and network bandwidth, potentially limiting efficiency on heterogeneous or cloud-based clusters. The evaluation is constrained to ~8B parameters and traditional NLP benchmarks, lacking analysis on instruction tuning, multimodal objectives, or extreme-scale regimes that emerged shortly after publication.
Broader impact
This work fundamentally reshaped the landscape of large-scale machine learning by providing the first practical, widely adoptable blueprint for training multi-billion parameter language models. It directly enabled the rapid scaling of LLMs, influencing nearly all subsequent distributed training frameworks (e.g., DeepSpeed, FairScale, JAX ecosystems) and democratizing access to large-model research. By establishing efficient scaling as a tractable engineering problem, it accelerated breakthroughs across NLP, code synthesis, and multimodal AI, while simultaneously catalyzing critical discourse around compute equity, energy consumption, and the centralization of frontier AI capabilities. The paper introduces a highly efficient tensor parallelism strategy for transformers that enabled the practical training of multi-billion parameter language models, fundamentally reshaping the ML systems landscape and serving as the foundational blueprint for modern large-scale model training.
Curator note
Shoeybi et al.; NVIDIA; tensor parallelism; standard multi-GPU training
Author of this paper? Send a correction.