PaLM: Scaling Language Modeling with Pathways
Abstract
Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
PaLM demonstrates that scaling a standard Transformer to 540B parameters yields discontinuous improvements in few-shot reasoning, code generation, and multilingual understanding, fundamentally shifting the field toward compute-driven capability scaling. The paper's primary contribution is not architectural novelty but rather a rigorous, large-scale empirical demonstration of scaling laws, emergent abilities, and comprehensive benchmark evaluation, establishing methodological standards and empirical baselines that have defined LLM research and development for years.
Methodology
The paper leverages a standard dense Transformer architecture scaled to 540B parameters, with the primary methodological innovation residing in the training infrastructure rather than the model itself. The Pathways system enables highly efficient, sparse routing of computation across 6144 TPU v4 chips, allowing stable training at unprecedented scale. The experimental methodology is rigorously structured around scaling laws, systematically evaluating few-shot, one-shot, and zero-shot regimes across >100 benchmarks. While the architecture is architecturally incremental, the systematic isolation of scale as the primary variable and the introduction of discontinuous improvement analysis ("emergent abilities") represent a mature, well-controlled empirical framework.
Results & evaluation
Exceptionally comprehensive and influential. The evaluation spans English NLP, multi-step reasoning (including chain-of-thought prompting), code generation, machine translation, and multilingual QA. The introduction of detailed BIG-bench analysis, where performance is tracked across multiple model sizes, provides the first large-scale empirical evidence of phase-transition-like scaling behavior. The paper also includes rigorous analyses of dataset contamination, memorization, and bias/toxicity, setting a new standard for responsible scaling evaluations. The breadth, statistical rigor, and task coverage are exemplary.
Reproducibility
Low to moderate for direct replication. The 540B model requires proprietary hardware (TPU v4 pods) and the Pathways distributed training system, neither of which are publicly available. Model weights and training code were not released, preventing independent verification of exact results. However, the scaling trends, benchmark protocols, and prompting methodologies are fully documented and have been widely adopted and reproduced in principle by other well-resourced labs.
Limitations
The dense architecture is computationally and memory-inefficient compared to subsequent compute-optimal (Chinchilla) and sparse/MoE approaches. The paper heavily emphasizes scale as the primary driver of capability, under-exploring data quality, curriculum learning, and architectural efficiency. The lack of open release limits community validation and downstream research. Ethical and safety analyses, while present, are observational rather than prescriptive, and do not address alignment or robust adversarial testing in depth.
Broader impact
Foundational to the modern LLM paradigm. The paper empirically cemented the "scale-first" hypothesis, catalyzing massive industry and academic investment in larger models. It introduced the concept of emergent abilities to mainstream ML discourse, reshaping how researchers evaluate and anticipate model capabilities. The comprehensive bias, memorization, and contamination analyses established baseline evaluation protocols that continue to inform safety, data curation, and responsible AI research. PaLM demonstrates that scaling a standard Transformer to 540B parameters yields discontinuous improvements in few-shot reasoning, code generation, and multilingual understanding, fundamentally shifting the field toward compute-driven capability scaling. The paper's primary contribution is not architectural novelty but rather a rigorous, large-scale empirical demonstration of scaling laws, emergent abilities, and comprehensive benchmark evaluation, establishing methodological standards and empirical baselines that have defined LLM research and development for years.
Curator note
Chowdhery et al.; Google; 540B params; chain-of-thought abilities
Author of this paper? Send a correction.