Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning
Abstract
RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp. Recent work has studied policy entropy as both a diagnostic and a control signal for this behavior. In our experiments, however, increasing entropy alone does not reliably change the eventual plateau. We analyze RL through its implicit target policy and derive a sharpness factor $κ$, together with an effective counterpart $κ_{\mathrm{eff}}$, that describes how strongly reward or advantage gaps are translated into probability-ratio gaps. Motivated by this view, we introduce \emph{reward budgeting}, a plug-in advantage construction that assigns a fixed total reward budget to each prompt. For binary rewards and independent on-policy rollouts, its unclipped score-function estimator follows the gradient of an exact finite-$G$ concave objective with diminishing returns, while its advantage gap decreases as more rollouts become correct. For general rewards, a shifted log-mean interpretation holds asymptotically under the stated regularity conditions. These results motivate, rather than establish exact objective equivalence for, the clipped implementations. On Qwen3-8B and Llama3.2-3B-Instruct, reward budgeting consistently improves a range of RL baselines, with gains up to +12.50 on AMC23 and +10.00 on AIME24 and AIME25. Overall, our results support an update-dynamics view of premature convergence and show that strong gains can be obtained without explicitly preserving entropy.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[The paper introduces reward budgeting, a principled advantage construction method that mitigates premature convergence in RL for LLM reasoning by regulating the implicit sharpening pressure of policy updates, demonstrating significant performance gains without explicit entropy control.]
Methodology
The paper proposes "reward budgeting," a plug-in advantage construction for RL that allocates a fixed total reward budget per prompt across rollouts based on their positive reward gaps. The core theoretical contribution is the derivation of a "sharpness factor" $\kappa$ from the KL-regularized RL objective, which characterizes how strongly reward gaps are amplified into probability-ratio gaps in the implicit target policy. The authors argue that premature convergence is driven by excessive sharpening pressure rather than entropy collapse per se. They demonstrate that reward budgeting implicitly regulates this sharpness factor, leading to a finite-$G$ concave objective with diminishing returns for binary rewards. The method is presented as a principled alternative to explicit entropy regularization, offering a mechanism to sustain learning signals without directly manipulating policy entropy.
Results & evaluation
Experiments are conducted on Qwen3-8B and Llama3.2-3B-Instruct using the DAPO-Math-17K dataset. The method is applied to various baselines (GRPO, DrGRPO, GSPO, DAPO, ClipCov, KLCov). Results show consistent improvements in final performance, with gains up to +12.50 on AMC23 and +10.00 on AIME24/25. The paper includes ablation studies comparing the effect of reward budgeting versus KL penalty removal, showing that gains are primarily attributable to the reward budgeting mechanism. Training dynamics analysis confirms that entropy often decreases while performance improves, supporting the claim that entropy preservation is not necessary.
Reproducibility
The paper provides detailed algorithmic descriptions (Algorithm 1) and hyperparameters (learning rate, batch size, rollout count). However, the code is stated to be released "upon approval," which may delay reproducibility. The specific filtering criteria for the training data and evaluation protocols are described, aiding reproducibility. The use of standard models and datasets (Qwen, Llama, DAPO-Math) is favorable for reproduction.
Limitations
The experiments are limited to verifiable reasoning tasks with predominantly binary rewards. The general-reward analysis relies on asymptotic assumptions that may not hold for noisy or open-ended rewards. The study uses a single training seed for main configurations, which limits statistical robustness. The group size $G=8$ is fixed, and no sweep over group sizes is provided. The theoretical results for general rewards are asymptotic and do not establish exact finite-$G$ objectives.
Broader impact
The paper contributes to the understanding of RL training dynamics for LLMs by shifting focus from entropy to sharpening pressure. The proposed reward budgeting technique is simple and plug-and-play, potentially widely adoptable in RLVR pipelines. The insights into the relationship between advantage construction and policy sharpening could inform the design of more stable and effective RL algorithms for LLM reasoning. [The paper introduces reward budgeting, a principled advantage construction method that mitigates premature convergence in RL for LLM reasoning by regulating the implicit sharpening pressure of policy updates, demonstrating significant performance gains without explicit entropy control.]
Author of this paper? Send a correction.