Long-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions
Abstract
Despite recent advances in policy learning, long-horizon manipulation remains difficult because learned policies must avoid compounding errors while preserving future feasibility. While search-based planning can explicitly reason over such future consequences, it quickly becomes expensive in high-dimensional continuous action spaces. This raises a central question: how can long-horizon reasoning from search be transferred into policies in continuous action spaces? To this end, we propose DISCO: Distilled Tree Search over Continuous Actions. DISCO uses progressive-widening MCTS in physics simulation to generate long-horizon plans. During search, a learned policy proposes continuous actions, while a learned value function scores actions by their expected progress toward task completion. Using GPU-parallelized simulation, DISCO periodically distills MCTS-generated plans into the policy-value model, improving the action proposals and value estimates used in subsequent searches. Finally, the learned policy-value model can be directly deployed without the need for simulator-based search at test time. Across simulated and real-world rearrangement tasks, DISCO outperforms behavior cloning, reinforcement learning, and search-based baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces DISCO, a method that distills long-horizon reasoning from GPU-parallelized MCTS into a deployable policy-value model for continuous action spaces. It represents a strong contribution to robot manipulation by effectively bridging the gap between sample-efficient planning and scalable policy learning, offering a viable solution to the compounding error problem in long-horizon tasks.
Methodology
The paper proposes DISCO, a framework that bridges the gap between sample-efficient search-based planning and scalable policy learning in continuous action spaces. The core innovation is the use of progressive-widening Monte Carlo Tree Search (MCTS) within a physics simulator, where a learned policy proposes actions and a learned value function scores them. Crucially, the system employs a distillation loop: MCTS-generated plans are periodically distilled back into the policy-value model. This iterative refinement allows the policy to internalize the long-horizon reasoning capabilities of the search process. The method relies on GPU-parallelized simulation to make the search tractable, which is a significant engineering contribution. The final deployed model is a standard policy-value network that operates without the simulator, offering a clear advantage in real-world deployment latency and robustness compared to online search methods.
Results & evaluation
The evaluation covers both simulated and real-world multi-object rearrangement tasks. The paper claims to outperform behavior cloning (BC), reinforcement learning (RL), and search-based baselines. The inclusion of real-world experiments is a strong positive, as many papers in this domain remain purely simulated. The comparison against search-based baselines is particularly relevant, as it isolates the benefit of the distillation process over raw search performance. However, the specific metrics for "long-horizon" success rates and the degree of error compounding reduction would need to be scrutinized in the full results section to confirm the magnitude of improvement.
Reproducibility
The method relies on specific hyperparameters for the MCTS (progressive widening factors, exploration constants) and the distillation frequency. The use of GPU-parallelized simulation suggests a dependency on specific hardware or simulation frameworks (likely Isaac Gym or similar), which may limit immediate reproducibility for labs without such infrastructure. The project page is provided, which likely contains code, aiding reproducibility.
Limitations
The primary limitation is the dependency on a high-fidelity physics simulator for the training phase. If the simulator-to-real-world gap is significant, the distilled policy may suffer from performance degradation, although the real-world results suggest this is mitigated. Additionally, the computational cost of GPU-parallelized MCTS during training can be substantial, potentially limiting the scalability to extremely high-dimensional action spaces or very long horizons beyond what was tested.
Broader impact
This work has significant implications for the field of robot learning, particularly for tasks requiring precise, long-horizon manipulation. By demonstrating that search-based reasoning can be effectively distilled into a reactive policy, it offers a pathway to combining the robustness of planning with the speed of learned policies. This could influence future approaches to hierarchical reinforcement learning and model-based control. The paper introduces DISCO, a method that distills long-horizon reasoning from GPU-parallelized MCTS into a deployable policy-value model for continuous action spaces. It represents a strong contribution to robot manipulation by effectively bridging the gap between sample-efficient planning and scalable policy learning, offering a viable solution to the compounding error problem in long-horizon tasks.
Author of this paper? Send a correction.