GoalDex: Learning Image-Goal-Conditioned Policies for Dexterous In-Hand Manipulation
Abstract
Dexterous in-hand manipulation is commonly formulated around explicit object pose estimation, yet reliable pose tracking is challenging under severe hand-object occlusions and often requires object-specific supervision. This dependence creates a bottleneck between perception and control that can limit robustness and scalability. We present an end-to-end visuomotor framework for dexterous in-hand reorientation that operates on RGB observations and proprioception without intermediate pose estimation. Our approach jointly learns visual representations and control policies through behavior distillation from privileged state teachers and auxiliary reconstruction objectives. Furthermore, we introduce image-goal-conditioned reorientation, allowing desired object configurations to be specified directly through images rather than canonical pose representations. We evaluate our approach across a diverse set of object geometries and instances in simulation, where it consistently outperforms pose-based methods and policies built on pretrained visual encoders. Finally, we demonstrate direct sim-to-real transfer of a single image-goal-conditioned policy across five real-world objects, showing that accurate dexterous reorientation can be achieved without explicit pose estimation. For videos and code release, please refer to the project website: https://rffr.leggedrobotics.com/works/goaldex/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GoalDex introduces an end-to-end visuomotor framework for dexterous in-hand manipulation that eliminates the need for explicit object pose estimation by jointly learning visual representations and control policies through behavior distillation and image-goal conditioning. The paper demonstrates robust sim-to-real transfer across diverse objects, offering a scalable and robust alternative to traditional pose-based pipelines in dexterous robotics.
Methodology
The paper proposes GoalDex, an end-to-end visuomotor framework for dexterous in-hand manipulation that bypasses explicit object pose estimation. The core methodological contribution is the use of behavior distillation from privileged state teachers combined with auxiliary reconstruction objectives to learn robust visual representations directly from RGB and proprioception. A significant innovation is the introduction of image-goal conditioning, where desired object configurations are specified via images rather than canonical pose vectors. This allows for more flexible goal specification and leverages the visual modality for both state and goal representation. The integration of a Vision-Language Model (VLM) to select visual goals from natural language instructions further extends the utility of the framework, bridging high-level semantic commands with low-level dexterous control.
Results & evaluation
The evaluation is conducted primarily in simulation across a diverse set of object geometries, demonstrating consistent outperformance over pose-based methods and policies using pretrained visual encoders. A critical strength of the experimental design is the direct sim-to-real transfer of a single image-goal-conditioned policy to five distinct real-world objects. The zero-shot transfer capability without explicit pose estimation in the real world is a strong empirical result, validating the robustness of the learned representations. The comparison against baseline methods that rely on explicit pose tracking highlights the practical advantages of the proposed approach in scenarios where pose estimation is unreliable due to occlusions.
Reproducibility
The paper mentions code release via the project website. The methodology relies on standard deep learning techniques (distillation, reconstruction) applied to a specific robotics setup. While the specific hyperparameters and simulation environment details are likely in the appendix (not fully provided in the excerpt), the use of standard simulation-to-real transfer protocols and clear description of the training objectives (privileged teachers, reconstruction) provides a reasonable path for reproduction. The reliance on specific hardware (dexterous hand) may limit immediate reproducibility for labs without such setups, but the simulation code should be accessible.
Limitations
The primary limitation is the reliance on simulation for training, which requires careful domain randomization to achieve zero-shot transfer. The method is tested on a specific set of objects and a specific dexterous hand; generalization to significantly different hand morphologies or object categories beyond the tested set is not fully established. The use of a VLM for goal selection adds computational overhead and potential latency, which may not be suitable for all real-time applications. Additionally, the "privileged state teacher" setup requires access to ground truth state information during training, which is a standard assumption in sim-to-real but still a dependency.
Broader impact
This work has significant implications for the field of dexterous robotics by demonstrating that explicit pose estimation is not a strict prerequisite for high-performance manipulation. By shifting the burden from geometric perception to end-to-end visuomotor learning, it opens the door to more robust systems in unstructured environments. The image-goal conditioning paradigm could be adopted in other manipulation tasks, providing a more intuitive interface for specifying goals. The successful sim-to-real transfer reinforces the viability of simulation-based learning for complex dexterous tasks, potentially reducing the cost and time associated with real-world data collection. GoalDex introduces an end-to-end visuomotor framework for dexterous in-hand manipulation that eliminates the need for explicit object pose estimation by jointly learning visual representations and control policies through behavior distillation and image-goal conditioning. The paper demonstrates robust sim-to-real transfer across diverse objects, offering a scalable and robust alternative to traditional pose-based pipelines in dexterous robotics.
Author of this paper? Send a correction.