Doob-Based Generative Policy Optimization
Abstract
Diffusion and flow models provide expressive policy classes for robotic and sequential decision-making systems. However, their action likelihoods are generally intractable, limiting the direct use of widely adopted policy optimization methods such as PPO and GRPO. We propose Doob Policy Optimization (DoobPO), a framework that leverages Doob's $h$-transform for likelihood-free reinforcement learning fine-tuning of generative policies. DoobPO separates policy improvement from generative policy learning. It first learns a lightweight ratio model that represents the desired policy update, and then distills the resulting ratio-tilted target policy into the generative policy via conditional drift matching. This formulation applies uniformly to both diffusion and flow policies and supports a broad family of policy improvement rules, including PPO and GRPO. We prove that the practical conditional drift matching objective is gradient-equivalent to the ideal Doob drift matching objective. We further show that the induced diffusion process solves an endpoint-constrained stochastic optimal control problem. This establishes that it achieves the minimum-energy modification required to realize the desired policy tilt. Across MuJoCo continuous-control benchmarks, DoobPO achieves strong performance against competitive diffusion- and flow-policy baselines, showing a principled and practical route to RL fine-tuning for generative policies.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
DoobPO introduces a likelihood-free reinforcement learning framework using Doob's $h$-transform to enable policy optimization for diffusion and flow-based policies. The paper makes a substantial technical contribution by unifying policy improvement and generative learning through conditional drift matching, supported by rigorous theoretical proofs and strong empirical results on MuJoCo benchmarks, positioning it as a significant advancement in the integration of generative models with reinforcement learning.
Methodology
The paper proposes Doob Policy Optimization (DoobPO), a framework addressing the critical bottleneck of intractable likelihoods in diffusion and flow-based policies for reinforcement learning. By leveraging Doob's $h$-transform, the method decouples policy improvement from generative policy learning. It introduces a lightweight ratio model to represent policy updates and distills the resulting ratio-tilted target policy into the generative policy via conditional drift matching. The authors provide a rigorous theoretical foundation, proving that the practical conditional drift matching objective is gradient-equivalent to the ideal Doob drift matching objective and demonstrating that the induced diffusion process solves an endpoint-constrained stochastic optimal control problem. This establishes a principled, minimum-energy modification approach to realizing desired policy tilts, offering a unified formulation for both diffusion and flow policies.
Results & evaluation
The experimental evaluation is conducted on MuJoCo continuous-control benchmarks, comparing DoobPO against competitive diffusion- and flow-policy baselines. The results demonstrate strong performance, validating the practical utility of the proposed framework for RL fine-tuning of generative policies. While the benchmarks are standard in the field, the paper focuses primarily on continuous control, which may limit the generalizability of the findings to discrete or high-dimensional visual tasks without further evidence.
Reproducibility
The paper is accepted at NeurIPS 2026, a venue with high standards for reproducibility. A project page is provided, which likely contains code and additional details. The theoretical proofs and clear separation of the ratio model and drift matching components suggest that the method is well-defined and implementable. However, without access to the full code repository details in the provided text, specific hyperparameter choices and implementation nuances remain partially opaque.
Limitations
The primary limitation is the focus on MuJoCo continuous-control tasks, which may not fully capture the challenges of applying DoobPO to more complex, high-dimensional, or discrete action spaces. Additionally, the reliance on a lightweight ratio model introduces an approximation that could affect performance in highly stochastic or non-stationary environments. The computational overhead of maintaining and updating the ratio model alongside the generative policy is not extensively discussed in the provided abstract.
Broader impact
This work has significant potential to bridge the gap between expressive generative models and practical reinforcement learning. By enabling the use of standard policy optimization methods like PPO and GRPO with diffusion and flow policies, DoobPO could accelerate the adoption of generative models in robotics and sequential decision-making. The theoretical insights into stochastic optimal control and drift matching may also inspire new approaches in related fields such as optimal transport and control theory. DoobPO introduces a likelihood-free reinforcement learning framework using Doob's $h$-transform to enable policy optimization for diffusion and flow-based policies. The paper makes a substantial technical contribution by unifying policy improvement and generative learning through conditional drift matching, supported by rigorous theoretical proofs and strong empirical results on MuJoCo benchmarks, positioning it as a significant advancement in the integration of generative models with reinforcement learning.
Author of this paper? Send a correction.