Pushing Dexterity and Safety for In-Hand Object Pose Reaching That Works Remarkably Well
Abstract
Dexterous manipulation with multi-fingered hands is the next frontier of robotics, yet learned hand skills remain limited to simple grasps or basic repeated motions, such as rotating an object. As robots attempt more demanding in-hand maneuvers, safety, such as not dropping the object, becomes critical, yet it is rarely measured. We study in-hand object pose reaching, a task that demands diverse and difficult grasps: moving an object to almost any position and orientation within reach of the fingers, at any wrist orientation, through long sequences of consecutive goals. We present a recipe for training such policies for a LEAP hand in simulation without human data, built on a range of insights, from diverse resets and goals via generated grasp datasets to an adaptive friction curriculum. With careful system identification, the policies transfer zero-shot to the real hand, using either motion capture or a single wrist-mounted depth camera, and prove remarkably capable and robust. On top of the task policy, we add a learned residual safety filter with adjustable conservativeness, letting us tune the balance between performance and safety and greatly reduce object drops. In 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and diverse wrist orientations, the state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad, and the filter cuts its drops by 75% while still reaching 91.5% of goals. The policies can also spin a lightbulb held in an arbitrary pose, reach out-of-distribution goals, withstand fast arm motion and perturbations, and, needing only a wrist camera, serve as portable skills such as screwing in a lightbulb. We release the full recipe as a platform for studying performance, reliability, and safety in high-dimensional dexterous systems. Videos and more are available at https://ian-chuang.github.io/dexsafe/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.
Methodology
The paper proposes a comprehensive sim-to-real pipeline for in-hand object pose reaching using a 16-DoF LEAP hand. The core methodological contributions are: (1) A training recipe that leverages a large-scale generated grasp dataset (via LightningGrasp and DexterityGen) to define diverse goals and resets, avoiding the need for human demonstrations. (2) An adaptive friction curriculum that starts with high friction to stabilize learning and gradually reduces it to match reality, which is shown to be critical for convergence. (3) A modular safety approach using a learned residual safety filter (adapted from ResSafe) that operates on top of a frozen task policy. This filter uses a tunable weight $w$ to balance performance and safety, allowing for a "dial" to adjust conservativeness at deployment without retraining the task policy. The system identification is rigorous, using a voltage-driven DC motor model (BAM) and precise latency calibration to bridge the sim-to-real gap.
Results & evaluation
The experimental evaluation is extensive and rigorous, particularly for a preprint. The authors conduct 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and 6 wrist orientations. The state-based policy achieves a 95.2% success rate, and the safety filter reduces drops by 75% while maintaining a 91.5% success rate. The paper also evaluates robustness to out-of-distribution goals, fast arm motion, and physical perturbations. Comparisons with Hamilton-Jacobi (HJ) safety filters demonstrate the superiority of the learned residual approach in terms of computational efficiency and performance trade-offs. The inclusion of a depth-camera-only policy that transfers zero-shot is a strong validation of the visual sim-to-real gap mitigation.
Reproducibility
The paper provides high reproducibility. It details the hardware setup, system identification process, simulation parameters, and training hyperparameters. The authors commit to releasing the full recipe, code, and trained checkpoints. The use of standard tools like MuJoCo Warp and PPO, along with clear descriptions of the reward functions and curricula, makes it feasible for other researchers to replicate the results. The specific details on latency calibration and actuator modeling are particularly valuable for the robotics community.
Limitations
The policies are trained per object, which limits generalization to new objects without retraining. The depth policies still show a visual sim-to-real gap on certain objects (knife, wineglass). The safety filter is also trained per object and per nominal policy. The downstream tasks (e.g., screwing in a lightbulb) rely on hard-coded arm motions, limiting the autonomy of the system. The evaluation is limited to a specific hand (LEAP) and arm (UR5), so generalizability to other hardware is not tested.
Broader impact
This work significantly advances the state of the art in dexterous manipulation by addressing the critical issue of safety and reliability in high-dimensional contact-rich systems. The modular safety filter approach is likely to be adopted in other robotic domains where safety is paramount. The comprehensive evaluation framework for measuring performance, reliability, and safety provides a valuable benchmark for the field. The ability to operate with only a wrist-mounted camera expands the practical applicability of such systems beyond controlled lab environments. The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.
Author of this paper? Send a correction.