Learning to summarize from human feedback
Abstract
As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces a scalable reinforcement learning from human feedback (RLHF) pipeline that significantly outperforms supervised and metric-based baselines in text summarization, establishing the foundational methodology for modern large language model alignment and demonstrating that direct optimization of human preferences yields qualitatively superior, more robust, and transferable generative models. The work rigorously bridges the gap between proxy optimization and human intent, providing a reproducible framework, a valuable open dataset, and critical empirical insights into reward modeling and over-optimization that have fundamentally reshaped how the field approaches LLM training, evaluation, and safety.
Methodology
The paper introduces a systematic three-stage pipeline for aligning language models with human intent: (1) supervised fine-tuning (SFT) on a curated dataset, (2) training a reward model (RM) on human pairwise comparisons using a Bradley-Terry objective, and (3) optimizing a generation policy via PPO with an explicit KL-divergence penalty against the SFT initialization. The methodology is rigorously engineered to address known failure modes in RL for text generation, particularly distributional shift and reward hacking. The KL penalty serves as both an entropy regularizer and a trust-region constraint, preventing catastrophic policy collapse. The human evaluation protocol is exceptionally well-designed, featuring iterative labeler calibration, confidence-weighted comparisons, and strict researcher-labeler agreement monitoring (77%), directly addressing the misalignment pitfalls documented in prior human feedback studies.
Results & evaluation
The experimental design is comprehensive and empirically robust. The authors demonstrate that RLHF policies significantly outperform both SFT baselines (including models 10x larger) and human-written references in pairwise preference evaluations on the Reddit TL;DR dataset. The analysis carefully controls for confounding factors like summary length, revealing that quality gains persist even after length normalization. Cross-domain transfer to CNN/DM news articles without task-specific fine-tuning is convincingly demonstrated, highlighting the RM's ability to learn generalizable quality signals. The paper also provides critical ablation studies on reward model scaling (data/model size), over-optimization dynamics (Goodhart's law in learned rewards), and the failure of traditional metrics like ROUGE to correlate with human judgment as model quality improves.
Reproducibility
High. The authors release a substantial dataset of 64,832 human comparisons, inference code for 1.3B models, detailed hyperparameters, and explicit training procedures. The compute requirements (~320 GPU-days for the 6.7B RL run) and human labeling costs are transparently documented, which, while a practical barrier for smaller labs, does not hinder methodological reproducibility. The step-by-step pipeline, open-sourced components, and clear architectural specifications enable well-resourced researchers to replicate and extend the work.
Limitations
The primary limitation is the substantial computational and human annotation overhead, making iterative scaling expensive and potentially prohibitive for resource-constrained groups. The reward model exhibits a measurable bias toward longer summaries and can be over-optimized to the point of preference divergence, indicating inherent fragility in learned reward landscapes. The dataset's domain skew (heavily weighted toward Reddit relationship/advice subreddits) and the demographic homogeneity of labelers (predominantly White/American) raise concerns about generalization and value alignment across diverse populations. Finally, the method assumes humans can reliably compare outputs, which may break down for highly technical, multi-step, or safety-critical tasks.
Broader impact
This work establishes the foundational paradigm for preference-based alignment, directly catalyzing the development of instruction-tuned and conversational LLMs that dominate modern AI. By demonstrating that direct optimization of human preferences yields qualitatively superior and more transferable models than proxy metrics or maximum likelihood training, it shifts the field toward intent-aligned objectives. The paper responsibly addresses dual-use risks, noting the potential for malicious fine-tuning (e.g., persuasive manipulation or toxic content generation) and emphasizing the need for inclusive labeling and careful objective specification. It also highlights the socioeconomic implications of automating complex cognitive tasks, advocating for proactive policy considerations around workforce displacement. This paper introduces a scalable reinforcement learning from human feedback (RLHF) pipeline that significantly outperforms supervised and metric-based baselines in text summarization, establishing the foundational methodology for modern large language model alignment and demonstrating that direct optimization of human preferences yields qualitatively superior, more robust, and transferable generative models. The work rigorously bridges the gap between proxy optimization and human intent, providing a reproducible framework, a valuable open dataset, and critical empirical insights into reward modeling and over-optimization that have fundamentally reshaped how the field approaches LLM training, evaluation, and safety.
Curator note
Stiennon et al.; OpenAI; early RLHF demonstration on summarization
Author of this paper? Send a correction.