Controlling Sharpening in LLM Post-Training
Abstract
LLM post-training via reinforcement learning has shown consistent improvements in accuracy at the cost of reducing behavioral coverage - a phenomenon known as sharpening that can stall progress on difficult tasks. Repeated sampling objectives such as $\mathrm{pass@k}$ and $\mathrm{max@k}$ are known to mitigate sharpening, but the mechanism behind it remains poorly understood. Moreover, existing approaches for $\mathrm{@k}$ optimization use dedicated algorithms, and no general framework exists for adapting established policy optimization methods from $\mathrm{@}1$ to $\mathrm{@k}$. We derive the optimal policies for KL-regularized $\mathrm{pass@k}$ and $\mathrm{max@k}$ objectives in closed form. Our explicit solutions reveal that $\mathrm{@k}$ optimization naturally redistributes learning pressure based on prompt difficulty, directing it toward harder problems. Crucially, we show that standard $\mathrm{@}1$ objectives achieve these exact same optimal policies through prompt-adaptive regularization or reward transformation. Ultimately, this yields an optimizer-independent explanation for why $\mathrm{@k}$ objectives reduce sharpening on easy tasks while prioritizing success on harder ones. Building on these insights, we introduce $\mathrm{@k}$ Adaptive Pressure for Policy Optimization (KAPPO). Following a one-time reference calibration, KAPPO seamlessly adapts existing $\mathrm{@}1$ methods to $\mathrm{@k}$ via prompt-adaptive regularization or reward transformation, without otherwise altering their training procedures. Experiments with several RL methods on mathematical reasoning and code generation demonstrate that KAPPO provides direct control over sharpening. Crucially, we demonstrate that $\mathrm{@k}$ pressure can be fully realized with single completion training, matching the performance of multi-completion $\mathrm{@k}$ optimizers at a fraction of the cost.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper derives closed-form optimal policies for KL-regularized $pass@k$ and $max@k$ objectives, revealing that they act as prompt-adaptive regularization that mitigates sharpening in LLM post-training. By introducing KAPPO, a framework that adapts standard $@1$ optimizers to $@k$ settings via single-completion training, the authors provide a cost-effective and theoretically grounded solution to the diversity-accuracy trade-off, demonstrating significant improvements in robustness and generalization on mathematical reasoning and code generation tasks.
Methodology
The paper presents a rigorous theoretical analysis of KL-regularized $pass@k$ and $max@k$ objectives. The core contribution is deriving the closed-form optimal policies for these objectives, revealing that they act as prompt-adaptive regularization or reward transformations. Specifically, it shows that $@k$ optimization naturally increases the effective regularization strength for easy prompts (where the reference model already succeeds) and decreases it for hard prompts. This provides a mechanistic explanation for why $@k$ objectives mitigate "sharpening" (the loss of behavioral diversity in LLMs during RL post-training). The proposed method, KAPPO, leverages this insight to adapt standard $@1$ optimizers (like DPO, A*PO, GRPO) to $@k$ settings without requiring expensive online sampling of $k$ completions. Instead, it uses a one-time reference calibration to estimate prompt-dependent parameters, allowing single-completion training to achieve the benefits of multi-completion training. The mathematical derivations are sound, utilizing variational arguments and concavity properties of the reward functionals.
Results & evaluation
The experiments are well-designed to validate the theoretical claims. The authors use two distinct model families (OLMo-3-7B-Instruct-SFT and Qwen3-4B-Base) to ensure generality. They evaluate on mathematical reasoning (binary rewards) and code generation (continuous rewards). The key experimental result is that KAPPO with $k>1$ maintains performance at larger evaluation budgets ($pass@k_{eval}$) compared to standard $@1$ training, which typically suffers from sharpening (performance drops as $k$ increases). Furthermore, the paper demonstrates that single-completion KAPPO matches the performance of dedicated multi-completion baselines (like PKPO) at a fraction of the computational cost. The use of AIME 2026 and LiveCodeBench v6 as out-of-distribution/harder test sets strengthens the claim that the method improves robustness and generalization rather than just overfitting to the training distribution.
Reproducibility
The paper provides a strong reproducibility statement. It claims that all datasets and models are publicly available. The method is described as a "minimalist intervention," requiring only a one-time calibration step and simple modifications to the loss function or regularization coefficient. Pseudocode for the calibration is mentioned in the appendix, and the codebase is stated to be available in supplementary materials. The hyperparameters and experimental details are described as being provided in the appendix. This high level of detail and the simplicity of the proposed modification make the work highly reproducible.
Limitations
The primary limitation is the reliance on a static bank of reference completions for calibration. If the reference model's distribution shifts significantly or if the task distribution is highly non-stationary, the pre-computed adaptive parameters might become suboptimal. Additionally, for continuous rewards, the method uses a win-rate surrogate rather than the raw $max@k$ objective, which is an approximation. The paper also notes that the win-rate solution assumes a continuous reference reward distribution, which may not hold for discrete rewards with ties. Finally, the experiments are limited to 4B-7B scale models; the scalability of the calibration cost to larger models or very large prompt sets is not extensively explored.
Broader impact
This work has significant implications for the field of LLM post-training. By providing a theoretical foundation for why $@k$ objectives work and a practical, low-cost way to implement them, it enables practitioners to better control the trade-off between accuracy and diversity in RL fine-tuning. This is crucial for developing LLMs that are not only accurate but also robust and capable of diverse reasoning strategies. The method's compatibility with existing optimizers like DPO and GRPO makes it easy to adopt, potentially leading to widespread use in industry and academia. It addresses a critical bottleneck in current RLVR (Reinforcement Learning with Verifiable Rewards) pipelines, where sharpening often limits further progress. The paper derives closed-form optimal policies for KL-regularized $pass@k$ and $max@k$ objectives, revealing that they act as prompt-adaptive regularization that mitigates sharpening in LLM post-training. By introducing KAPPO, a framework that adapts standard $@1$ optimizers to $@k$ settings via single-completion training, the authors provide a cost-effective and theoretically grounded solution to the diversity-accuracy trade-off, demonstrating significant improvements in robustness and generalization on mathematical reasoning and code generation tasks.
Author of this paper? Send a correction.