Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data
Abstract
Large language models trained with Reinforcement Learning (RL) with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning (SFT) are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper challenges the dominant view that RL is superior to SFT for generalization by demonstrating that SFT, when equipped with diverse and sufficiently scaled data, can match or exceed RL performance. It provides both empirical evidence on modern LLMs and theoretical insights into the training dynamics, offering a nuanced understanding of the role of data in post-training.
Methodology
The paper employs a rigorous comparative study design, isolating the variables of data source and data scale to challenge the prevailing narrative that RL is inherently superior to SFT for generalization. The methodology involves training Qwen2.5 and Qwen3 models under controlled conditions, specifically matching the number of correct rollouts and total rollout budgets between SFT and RL setups. A key methodological strength is the use of LLM annotations to characterize solution methods, allowing the authors to link dataset diversity (specifically "tail methods") to generalization performance. The inclusion of theoretical analysis of shallow transformers provides a mechanistic explanation for the empirical findings, bridging the gap between heuristic observation and theoretical understanding.
Results & evaluation
The experiments are extensive, utilizing widely adopted model families (Qwen2.5/3) which enhances the practical relevance of the findings. The evaluation metrics focus on generalization, likely measured via out-of-distribution tasks or complex reasoning benchmarks. The finding that SFT can match or exceed RL when data scale and diversity are properly controlled is a significant empirical result. The comparison is fair by matching rollout budgets, which is a critical control often missing in such comparisons. The results suggest that the "memorization" of SFT is largely an artifact of insufficient data diversity and scale rather than an inherent limitation of the training paradigm.
Reproducibility
The paper appears to be highly reproducible given the use of open-source model architectures (Qwen) and standard training paradigms. However, the specific "carefully mixed SFT dataset" and the exact protocols for LLM annotation of solution methods would require detailed appendices to fully replicate. The theoretical proofs provided in the appendix support the validity of the shallow transformer analysis.
Limitations
The study is limited to the Qwen family of models, and it is unclear if these findings generalize to other architectures like Llama or Mistral. The theoretical analysis is restricted to shallow transformers, which may not fully capture the dynamics of deep, large-scale LLMs. Additionally, the cost of generating large-scale SFT data with diverse methods may be prohibitive for smaller labs, limiting the immediate practical adoption of the "SFT > RL" strategy.
Broader impact
This paper has significant implications for the post-training strategies of LLM developers. If SFT can achieve comparable or better generalization with careful data curation, it could reduce the computational overhead associated with RLHF/RLVR pipelines. It shifts the focus from the training algorithm to the quality and diversity of the training data, potentially leading to new standards for dataset construction in the LLM community. The paper challenges the dominant view that RL is superior to SFT for generalization by demonstrating that SFT, when equipped with diverse and sufficiently scaled data, can match or exceed RL performance. It provides both empirical evidence on modern LLMs and theoretical insights into the training dynamics, offering a nuanced understanding of the role of data in post-training.
Author of this paper? Send a correction.