Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation
Abstract
Video models provide rich priors for generalizable robotic manipulation, but plausible visual futures are not always executable. We introduce DreamCan, a hierarchical policy that grounds video models through sparse visual planning: a high-level planner adapted from a pre-trained video model and a low-level visuomotor policy. Rather than predicting dense trajectories, the high-level planner generates sparse visual subgoals that balance reachability and task utility. From offline, action-free robot videos, we estimate these quantities and use dynamic programming to select subgoal sequences without manual annotations. The resulting subgoals are more reliably attained than uniformly spaced alternatives, and the full system outperforms language, dense-video, and uniform-subgoal planners, reaching high planning success rates on Language-Table tasks and 91.2% success on LIBERO-Long with a 0.1B-parameter low-level policy. Additional suboptimal videos and test-time compute further improve success. The system also performs challenging real-world bimanual tasks, including box folding and packing with unseen object combinations. More results are available at https://dreamcan.github.io/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
DreamCan introduces a hierarchical policy that uses sparse visual subgoals from video models to enable long-horizon robot manipulation. The paper presents a rigorous evaluation showing that this approach outperforms dense and uniform planning baselines, achieving high success rates in simulation and real-world bimanual tasks, thereby demonstrating the viability of video-based sparse planning for complex robotic manipulation.
Methodology
The paper proposes "DreamCan," a hierarchical policy that bridges the gap between generative video models and executable robot actions. The core innovation lies in the high-level planner, which is adapted from a pre-trained video model to generate *sparse* visual subgoals rather than dense trajectories. This is a significant architectural choice, as dense video predictions are often physically infeasible for current robot hardware. The method estimates subgoal reachability and task utility from offline, action-free videos, utilizing dynamic programming to select optimal subgoal sequences without manual annotation. This unsupervised discovery of subgoals is a strong methodological contribution, reducing the reliance on expensive human demonstrations for long-horizon tasks. The low-level policy is a goal-conditioned visuomotor policy that executes the actions to reach these subgoals.
Results & evaluation
The experimental evaluation is robust, covering both simulation (Language-Table, LIBERO-Long) and real-world bimanual tasks. Achieving 91.2% success on LIBERO-Long with a small 0.1B-parameter low-level policy is a notable result, suggesting that the sparse planning approach effectively offloads complexity from the low-level controller. The comparison against language-based, dense-video, and uniform-subgoal planners provides a comprehensive baseline analysis. The inclusion of real-world bimanual tasks (box folding, packing) with unseen object combinations demonstrates generalization capabilities beyond simulation. The observation that additional suboptimal videos and test-time compute improve success rates is a valuable empirical finding for practitioners.
Reproducibility
The paper provides a project URL (dreamcan.github.io) which likely contains code and additional results. The methodology relies on pre-trained video models and standard dynamic programming, which are well-understood components. However, the specific adaptation of the video model for sparse subgoal generation and the estimation of reachability/utility from action-free videos may require careful implementation details that are not fully visible in the abstract. The use of offline data is a practical advantage for reproducibility in real-world settings where collecting new action data is costly.
Limitations
The reliance on pre-trained video models means the system's performance is bounded by the quality and coverage of the video priors. If the video dataset does not contain examples of the specific task or object configurations, the planner may fail. The method is designed for manipulation tasks; its applicability to locomotion or other robotic domains is not discussed. The computational cost of the high-level planner (video model inference) and the dynamic programming step could be a bottleneck for real-time operation, although test-time compute scaling is mentioned as a positive factor.
Broader impact
This work has significant implications for the field of robot learning, particularly for long-horizon manipulation tasks. By leveraging video models as priors for planning, it offers a path towards more generalizable and data-efficient robot policies. The sparse subgoal approach is a practical solution to the "plausibility vs. executability" problem in video-based planning. This could influence future research in hierarchical reinforcement learning and the integration of generative models into robotic control loops. DreamCan introduces a hierarchical policy that uses sparse visual subgoals from video models to enable long-horizon robot manipulation. The paper presents a rigorous evaluation showing that this approach outperforms dense and uniform planning baselines, achieving high success rates in simulation and real-world bimanual tasks, thereby demonstrating the viability of video-based sparse planning for complex robotic manipulation.
Author of this paper? Send a correction.