Training Compute-Optimal Large Language Models (Chinchilla)
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper establishes that model parameters and training tokens should scale equally with compute budget, fundamentally correcting prior scaling assumptions and enabling the development of smaller, more efficient, and higher-performing large language models. Through rigorous empirical scaling analysis across hundreds of training runs and a landmark 70B-parameter validation, the work provides a mathematically grounded, highly reproducible framework that has permanently shifted industry and academic LLM training practices toward compute-optimal data scaling, while highlighting the urgent need for high-quality dataset curation and responsible scaling ethics.
Methodology
The paper introduces a rigorous, multi-pronged empirical framework to derive compute-optimal scaling laws for autoregressive transformers. By deploying three complementary methodologies—(1) training envelope extraction across varied horizons, (2) IsoFLOP profiling at fixed compute budgets, and (3) parametric loss surface fitting with Huber-robust optimization—the authors systematically isolate the trade-off between parameter count ($N$) and training tokens ($D$). Crucially, they identify and correct a key methodological bias in prior scaling work (Kaplan et al., 2020): the use of fixed learning rate schedules and token counts, which artificially inflated the perceived marginal returns of parameter scaling. The derivation that $N \propto C^{0.5}$ and $D \propto C^{0.5}$ (i.e., equal scaling) is mathematically grounded in risk decomposition and empirically consistent across all three approaches and multiple datasets (The Pile, C4, GitHub code).
Results & evaluation
The experimental design is exceptionally thorough, encompassing over 400 controlled training runs spanning 70M to 16B parameters and 5B to 500B tokens. The large-scale validation (Chinchilla: 70B params, 1.4T tokens) directly tests the predicted frontier against Gopher (280B, 300B tokens) and other contemporary LLMs under identical compute budgets. Evaluation spans language modeling (Pile subsets), academic reasoning (MMLU), complex reasoning (BIG-bench), reading comprehension, closed-book QA, and safety/toxicity metrics. Chinchilla achieves state-of-the-art results across nearly all benchmarks while reducing inference memory and FLOPs by ~4x. The ablation on optimizers (AdamW vs Adam), learning rate schedules, and dataset composition further strengthens the empirical claims.
Reproducibility
High. The paper provides explicit FLOP accounting formulas, detailed hyperparameter tables (layers, heads, $d_{model}$, batch sizes, LR schedules), dataset sampling proportions, and loss decomposition equations. While no public code repository is linked, the methodological transparency, precise compute budgeting, and clear scaling formulas enable independent replication. The IsoFLOP methodology is particularly straightforward to implement in modern distributed training frameworks.
Limitations
The authors appropriately acknowledge several constraints: (1) extrapolation relies on power-law assumptions despite observing negative curvature in the FLOP-loss frontier at higher scales, suggesting optimal models may be even smaller than predicted; (2) analysis is restricted to single-epoch training, leaving multi-epoch dynamics and data reuse unexplored; (3) only two large-scale runs (Chinchilla and Gopher) exist for direct validation, creating a gap in intermediate-scale verification; (4) scaling to trillions of tokens exacerbates dataset quality, privacy, and toxicity risks, which are noted but not mitigated.
Broader impact
This work fundamentally redefined the LLM training paradigm, shifting the industry from parameter-centric scaling to compute-optimal data scaling. It directly enabled the development of highly efficient, high-performing models (e.g., Llama, Mistral, Qwen families) that achieve superior capabilities at a fraction of the inference cost. The paper also underscores the critical bottleneck of high-quality dataset curation and raises important ethical considerations regarding web-scale data collection, bias propagation, and privacy at trillion-token scales. Its methodological framework is now standard practice in both academic and industrial LLM development. This paper establishes that model parameters and training tokens should scale equally with compute budget, fundamentally correcting prior scaling assumptions and enabling the development of smaller, more efficient, and higher-performing large language models. Through rigorous empirical scaling analysis across hundreds of training runs and a landmark 70B-parameter validation, the work provides a mathematically grounded, highly reproducible framework that has permanently shifted industry and academic LLM training practices toward compute-optimal data scaling, while highlighting the urgent need for high-quality dataset curation and responsible scaling ethics.
Curator note
Hoffmann et al.; revised scaling laws; data matters as much as params
Author of this paper? Send a correction.