Foundation Pareto Flow Policy for Multi-Objective Reinforcement Learning
Abstract
The overarching goal of reinforcement learning (RL) is to efficiently optimize the agent's policy for various environments, tasks, dynamics, and even user preferences, which often consist of multiple and potentially conflicting rewards/objectives. This necessitates the learning of Pareto-optimal policies that can navigate different trade-offs among objectives. However, existing multi-objective RL (MORL) methods, whether trained online or offline, are typically limited to task- or objective-specific policy learning and do not address generalization of learned trade-offs to unseen tasks/objectives. In this work, we propose the Foundation Pareto Flow Policy (FP2) for offline MORL in continuous robotic control, inspired by the pre-mid-post-training paradigm underlying recent foundation models. Specifically, we instantiate FP2 as a multi-modal diffusion transformer equipped with a likelihood-free flow matching loss, enabling unified, scalable, and instructable learning of diverse trade-off trajectories. We further develop a three-stage training pipeline to gradually optimize the policy: (1) pre-training on a large-scale MORL dataset, (2) mid-training on a carefully curated high-quality dataset, and (3) post-training on few collected trajectory data from new tasks and trade-offs. Empirical results demonstrate that FP2 outperforms previous state-of-the-art methods on most tasks of the D4MORL benchmark, achieving superior zero-shot interpolation and extrapolation on complex Pareto manifolds as well as efficient few-shot adaptation of the learned trade-off prior to new continuous-control tasks. Code is available at https://github.com/FibonacciRabbit/FP2.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces FP2, a foundation policy for offline MORL that leverages a three-stage pre-mid-post training pipeline on a diffusion transformer backbone to achieve superior generalization across preferences and tasks. By adapting the foundation model training recipe to the specific challenges of multi-objective trade-offs—specifically using mid-training on curated Pareto-optimal data and a triplet loss for preference alignment—the method demonstrates significant improvements in zero-shot Pareto front coverage and few-shot cross-task transfer compared to state-of-the-art offline MORL methods. This work is significant as it provides a scalable, unified framework for learning reusable multi-objective priors, moving beyond task-specific policy learning and offering a path toward more adaptable robotic agents capable of handling dynamic user preferences and conflicting objectives.
Methodology
The paper proposes FP2, a foundation model approach for offline Multi-Objective Reinforcement Learning (MORL). The core methodological contribution is the application of the "pre-mid-post" training paradigm, popularized in Large Language Models, to the domain of continuous control policies. The architecture utilizes a Multi-Modal Diffusion Transformer (MMDiT) with a Rectified Flow objective. The "mid-training" stage is particularly interesting, as it fine-tunes on a curated subset of high-quality Pareto-optimal trajectories to sharpen preference-behavior alignment before final post-training on new tasks. The inclusion of a triplet velocity regularization loss to enforce monotonicity in preference space is a clever technical detail that addresses the specific challenge of continuous preference generalization.
Results & evaluation
The experiments are conducted on the D4MORL benchmark, which is a standard for offline MORL. The paper demonstrates strong performance in zero-shot interpolation/extrapolation of Pareto fronts and few-shot adaptation to new tasks. The ablation studies effectively isolate the contributions of the three training stages and the specific loss functions. The comparison against strong baselines like MODULI shows consistent improvements in Hypervolume and Preference Alignment Score. The cross-task transfer experiments (leave-one-out) provide strong evidence for the "foundation model" claim, showing that the learned trade-off prior transfers to unseen dynamics.
Reproducibility
The paper provides a GitHub link and detailed implementation notes in the appendix, including hyperparameters, dataset splits, and architectural specifics. The deterministic generation of text prompts from numerical preferences is clearly explained, aiding reproducibility. The use of standard benchmarks (D4MORL, Meta-World) further supports reproducibility.
Limitations
The primary limitation is the reliance on template-based text prompts, which limits the flexibility of the "instructable" aspect to free-form language. The evaluation is confined to state-based continuous control; the paper acknowledges that extending to vision-based inputs is future work. Additionally, the "foundation" claim is somewhat modest given the limited number of environments (6 MuJoCo tasks) compared to true foundation models in other domains.
Broader impact
This work bridges the gap between generative modeling (diffusion/flow matching) and multi-objective decision making. It offers a scalable framework for handling conflicting objectives in robotics, which is crucial for real-world deployment where safety, efficiency, and speed often conflict. The three-stage training pipeline could be adapted to other offline RL settings where data quality varies. The paper introduces FP2, a foundation policy for offline MORL that leverages a three-stage pre-mid-post training pipeline on a diffusion transformer backbone to achieve superior generalization across preferences and tasks. By adapting the foundation model training recipe to the specific challenges of multi-objective trade-offs—specifically using mid-training on curated Pareto-optimal data and a triplet loss for preference alignment—the method demonstrates significant improvements in zero-shot Pareto front coverage and few-shot cross-task transfer compared to state-of-the-art offline MORL methods. This work is significant as it provides a scalable, unified framework for learning reusable multi-objective priors, moving beyond task-specific policy learning and offering a path toward more adaptable robotic agents capable of handling dynamic user preferences and conflicting objectives.
Author of this paper? Send a correction.