Decoupling Exploration from Optimization in RLVR
Abstract
Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@$k$ scaling, indicating that ExpDis produces models that generate more diverse correct solutions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a decoupled exploration-distillation framework for RLVR that effectively boosts reasoning performance and diversity without degrading general capabilities. By separating the aggressive exploration policy from the stable optimization policy, it resolves the tension between novelty and correctness, offering a robust and scalable improvement over standard RLVR methods.
Methodology
The paper proposes Exploration-Distillation (ExpDis), a framework that decouples exploration from optimization in Reinforcement Learning with Verifiable Rewards (RLVR). The core insight is that applying novelty bonuses directly to the policy being optimized leads to degradation of general capabilities because the verifiable reward only supervises a narrow slice of the model's knowledge. By training separate "explorer" policies with aggressive novelty incentives (using Random Network Distillation) and distilling only their correct, high-quality trajectories into a "student" policy via SFT followed by standard RLVR, the method isolates the benefits of exploration from the risks of instability. The methodology is sound, building on established concepts like intrinsic motivation and expert iteration, but the specific application to LLM reasoning with the decoupled architecture is a novel and effective contribution.
Results & evaluation
The experiments are rigorous and well-controlled. The authors compare against strong baselines (DAPO, GRPO, Dr. GRPO) and a direct novelty-bonus variant of DAPO. They demonstrate that ExpDis outperforms DAPO at the same wall-clock budget across multiple model families (Qwen3, Ministral) and benchmarks (AIME, MATH, etc.). Crucially, they show that while direct novelty bonuses degrade general knowledge benchmarks (MMLU, GPQA), ExpDis preserves or improves these capabilities while boosting reasoning performance. The analysis of pass@$k$ scaling and diversity metrics provides strong evidence that the method genuinely expands the model's solution space rather than just sharpening existing modes.
Reproducibility
The paper provides high reproducibility. Code and checkpoints are released on GitHub and HuggingFace. Hyperparameters, including the novelty bonus weight schedule and filtering criteria, are detailed in the appendix. The use of standard datasets (DAPO-Math-17K) and public benchmarks further aids reproducibility.
Limitations
The method requires training multiple models (explorers + student), which increases memory and compute overhead compared to single-model RLVR, although the paper argues this is offset by the ability to use aggressive exploration. The current evaluation is limited to mathematical reasoning; it is unclear if the benefits generalize to other domains like code or open-ended generation without similar verifiable rewards. The optimal number of explorers and rounds may be sensitive to the specific task and model scale.
Broader impact
This work has significant implications for the scaling of LLM reasoning. As models become proficient at known strategies, the need for discovering new strategies grows. ExpDis provides a safe mechanism to do so, potentially enabling the discovery of novel reasoning paths that standard RLVR would miss due to entropy collapse. It offers a practical path for practitioners to enhance model diversity and capability without the risk of catastrophic forgetting or quality degradation. The paper introduces a decoupled exploration-distillation framework for RLVR that effectively boosts reasoning performance and diversity without degrading general capabilities. By separating the aggressive exploration policy from the stable optimization policy, it resolves the tension between novelty and correctness, offering a robust and scalable improvement over standard RLVR methods.
Author of this paper? Send a correction.