Machine Learning PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-16

Regularized Score Centering

Hao Bai, Chenlu Ye, Xuanchang Zhang, Aviral Kumar, Tong Zhang

University of Illinois Urbana-Champaign · arXiv

Affiliations: University of Illinois Urbana-Champaign

Abstract

Asynchronous reinforcement learning increases training throughput, but data from stale policies and numerical mismatch between training and inference can destabilize learning. To stabilize off-policy learning, score centering (SC) removes the effect of reward baselines on the expected update. Our gradient diagnostics show less extreme gradients with SC than with importance sampling (IS), but SC can still fail when the learner moves far from the behavior policy that generated the data. To limit this policy mismatch, we introduce Regularized Score Centering (RSC), which adds a reverse Kullback-Leibler (KL) penalty to encourage the learner's next-token distribution to stay close to the behavior policy's next-token distribution. RSC works with or without truncated importance sampling (TIS), which caps token importance weights. Our theoretical analysis supports this approach, showing that KL regularization improves convergence guarantees. We evaluate dense and mixture-of-experts models on mathematical reasoning and agentic tasks, including online shopping, tool use, games, and fully asynchronous software engineering. RSC with TIS substantially improves sample efficiency across all agentic benchmarks, attaining higher training rewards with the same number of sampled responses. In final evaluations, it achieves top performance across all six agentic tasks, improving by 14.3% on AppWorld and reaching 9.9 times the strongest baseline's score on BabyAI.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper introduces Regularized Score Centering (RSC) to stabilize asynchronous reinforcement learning by adding a reverse KL penalty to mitigate policy mismatch. The technical contribution is significant, offering both theoretical guarantees and strong empirical results across diverse agentic tasks, demonstrating improved sample efficiency and final performance.

Methodology

The paper proposes Regularized Score Centering (RSC), a method to stabilize asynchronous reinforcement learning by addressing policy mismatch. The core innovation is the addition of a reverse Kullback-Leibler (KL) penalty to the standard score centering objective. This penalty encourages the learner's next-token distribution to remain close to the behavior policy's distribution, thereby limiting the divergence that causes instability in off-policy settings. The authors provide theoretical analysis showing that this KL regularization improves convergence guarantees. The method is designed to be compatible with truncated importance sampling (TIS), offering flexibility in implementation. The approach is grounded in a clear diagnostic of the failure modes of standard score centering, specifically when the learner moves far from the behavior policy.

Results & evaluation

The experimental evaluation is extensive, covering both dense and mixture-of-experts models. The tasks include mathematical reasoning and a suite of agentic benchmarks such as online shopping, tool use, games, and fully asynchronous software engineering. The results demonstrate that RSC with TIS significantly improves sample efficiency, achieving higher training rewards with the same number of sampled responses. In final evaluations, the method achieves top performance across all six agentic tasks, with notable improvements like 14.3% on AppWorld and 9.9x the strongest baseline on BabyAI. The breadth of tasks and model types suggests robustness.

Reproducibility

The paper provides a project page (https://biechi.github.io/blog/rsc/) which likely contains code and further details. The methodology is clearly described, and the use of standard benchmarks (AppWorld, BabyAI) facilitates comparison. However, the full text provided is a summary/abstract with section headers, so specific hyperparameters and implementation details are not visible in the provided snippet, though the project page suggests availability.

Limitations

The primary limitation is the reliance on the behavior policy for the KL penalty, which may be computationally expensive if the behavior policy is complex or not easily accessible. The method is specific to asynchronous RL settings, so its applicability to synchronous or on-policy learning is not addressed. The theoretical guarantees, while present, may have assumptions that limit their practical scope.

Broader impact

This work has significant implications for the stability and efficiency of large-scale asynchronous RL training, which is crucial for training large language models and agents. By providing a principled way to handle policy mismatch, it could enable more reliable and faster training of complex agents, potentially accelerating research in autonomous systems and LLM alignment. The paper introduces Regularized Score Centering (RSC) to stabilize asynchronous reinforcement learning by adding a reverse KL penalty to mitigate policy mismatch. The technical contribution is significant, offering both theoretical guarantees and strong empirical results across diverse agentic tasks, demonstrating improved sample efficiency and final performance.

Author of this paper? Send a correction.