Scaling Laws for Neural Language Models
Abstract
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper establishes the first rigorous, predictive scaling laws for neural language models, demonstrating that performance follows precise power-law relationships with model size, dataset size, and compute. Through extensive empirical validation across seven orders of magnitude, it reveals that architectural details are secondary to scale, derives optimal compute allocation strategies favoring large models trained with early stopping, and provides a foundational framework that has directly guided the development of modern foundation models while sparking critical discourse on the sustainability and accessibility of compute-driven AI progress.
Methodology
The paper employs a rigorous, large-scale empirical methodology to characterize the relationship between language model performance (cross-entropy loss) and three primary scaling factors: model parameters (N), dataset size (D), and training compute (C). The authors systematically vary architectural hyperparameters (depth, width, attention heads) to demonstrate their secondary importance relative to scale. The core methodological contribution is the derivation of a joint power-law ansatz for L(N,D) and L(N,S), grounded in asymptotic limits and analyticity constraints. The introduction of the critical batch size adjustment (B_crit) to normalize compute efficiency across different training regimes is methodologically sound and bridges optimization theory with empirical scaling. While heavily empirical rather than theoretically derived from first principles, the mathematical formulation is elegant, internally consistent, and provides a predictive framework rather than mere curve-fitting.
Results & evaluation
The experimental scope is unprecedented for its time, spanning over seven orders of magnitude in compute, model sizes from ~700M to 1.5B parameters, and datasets up to 23B tokens. The authors conduct comprehensive ablations on architecture, context length, batch size, and data distribution, demonstrating robust power-law trends across diverse validation sets (WebText2, Books, Wikipedia, Common Crawl). The use of early stopping to isolate dataset bottlenecks and the careful tracking of gradient noise scale for batch size optimization reflect meticulous experimental design. Results are highly consistent, with minimal variance across seeds, and the extrapolation methodology is validated against held-out compute regimes. The empirical rigor sets a new standard for scaling studies in deep learning.
Reproducibility
High. The paper provides explicit equations, fitted exponents, and detailed training configurations (optimizer, learning rate schedules, batch sizes, tokenization, and compute estimation formulas). The scaling laws themselves have been extensively reproduced and validated by independent research groups and industry labs. While the exact WebText2 dataset and compute infrastructure are proprietary, the mathematical framework and training protocols are sufficiently detailed to replicate the scaling behavior on alternative corpora and architectures. Subsequent open-source efforts (e.g., Chinchilla, LLaMA scaling studies) have confirmed the core power-law relationships, validating the reproducibility of the central claims.
Limitations
The study is purely empirical and lacks a formal theoretical derivation explaining why power-laws emerge in high-dimensional optimization landscapes. The extrapolation assumes smooth, uninterrupted scaling, which later work (e.g., DeepMind's Chinchilla) demonstrated breaks down when optimizing the N/D ratio for compute efficiency, suggesting the original compute-optimal frontier overestimates the value of model size relative to data. The analysis is restricted to decoder-only Transformers trained with autoregressive maximum likelihood, ignoring the impact of instruction tuning, reinforcement learning, and multimodal objectives. Additionally, computational constraints capped experiments at 1.5B parameters, requiring extrapolation to predict behavior at modern scales (10B-100B+), where hardware bottlenecks, sparsity, and parallelism overheads introduce non-linearities not captured by the model.
Broader impact
This paper fundamentally reshaped the AI research and industry paradigm, shifting focus from architectural innovation to compute and parameter scaling as the primary driver of performance gains. It directly informed the training strategies behind GPT-3 and subsequent foundation models, accelerating the development of highly capable LLMs. However, it also catalyzed significant concerns regarding compute centralization, environmental impact, and the widening resource gap between well-funded industry labs and academic/open-source researchers. The paper's emphasis on scale over architectural efficiency has sparked ongoing debate about sustainable AI development and the need for algorithmic breakthroughs that decouple performance from exponential compute growth. This paper establishes the first rigorous, predictive scaling laws for neural language models, demonstrating that performance follows precise power-law relationships with model size, dataset size, and compute. Through extensive empirical validation across seven orders of magnitude, it reveals that architectural details are secondary to scale, derives optimal compute allocation strategies favoring large models trained with early stopping, and provides a foundational framework that has directly guided the development of modern foundation models while sparking critical discourse on the sustainability and accessibility of compute-driven AI progress.
Curator note
Kaplan et al.; power-law compute/data/parameter tradeoffs
Related later work
Author of this paper? Send a correction.