GUARD: Anomaly-Informed Q-Steering for Generative Robot Policies in Dexterous Manipulation
Abstract
Generative robot policies based on diffusion and flow matching can model complex, multimodal behaviors, but remain susceptible to covariate shift during closed-loop execution. Existing approaches mitigate such failures through additional demonstrations, policy fine-tuning, or test-time guidance, but often require human intervention, engineered rewards, or other task-specific signals. We introduce GUARD, an anomaly-informed Q-steering framework that leverages the policy's training distribution to construct a dense, failure-anticipatory steering signal. GUARD pairs sparse task rewards with dense anomaly scores learned from policy training data and propagates both through a learned Q-function. Its gradients then steer the denoising process toward actions that improve task return while preventing trajectories from entering poorly supported regions, without directly modifying the pretrained policy. Across four Adroit dexterous manipulation tasks, GUARD achieves average success rates of $96.0\%$ and $80.0\%$ under nominal and perturbed dynamics, outperforming the strongest baseline by $21.0$ and $20.0$ percentage points while maintaining the lowest anomaly score. GUARD further demonstrates direct sim-to-real transfer on in-hand screwdriver rotation and reorientation tasks, achieving $40$ and $25$ percentage-point gains in success rate. These results demonstrate that anomaly signals can serve not only for failure monitoring but also to complement sparse task rewards as effective supervision for robust policy steering.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GUARD introduces an anomaly-informed Q-steering framework that enhances the robustness of generative robot policies by using dense anomaly scores to guide the denoising process, achieving significant improvements in success rates for dexterous manipulation tasks and demonstrating effective sim-to-real transfer. The paper presents a compelling solution to the covariate shift problem in closed-loop execution, offering a test-time steering mechanism that complements sparse rewards with failure-anticipatory signals, thereby improving both nominal and perturbed performance without modifying the pretrained policy.
Methodology
The paper proposes GUARD, a framework that integrates anomaly detection with Q-learning to steer generative robot policies (diffusion/flow matching) at test time. The core innovation is the use of a dense anomaly score, derived from the policy's training distribution, as a regularizer within a Q-function. This Q-function combines sparse task rewards with the anomaly signal, and its gradients are used to guide the denoising process. This approach avoids modifying the pretrained policy weights, instead acting as a test-time steering mechanism. The methodology is sound, leveraging the fact that generative models can provide a likelihood proxy (via anomaly scores) to identify out-of-distribution states, which is a common failure mode in closed-loop robotics. The integration of this signal into a Q-function for gradient-based steering is a logical and effective extension of recent work on reward-guided diffusion policies.
Results & evaluation
The experiments are conducted on four Adroit dexterous manipulation tasks, a standard benchmark for high-dimensional control. The paper reports high success rates (96.0% nominal, 80.0% perturbed) and claims significant improvements over baselines (21.0 and 20.0 percentage points). Crucially, the paper includes sim-to-real transfer experiments on in-hand screwdriver rotation and reorientation, showing gains of 40 and 25 percentage points. The inclusion of real-world validation is a strong point, as it addresses the critical gap between simulation and physical deployment. The comparison with "strongest baseline" is vague without specific naming in the abstract, but the magnitude of improvement suggests a robust evaluation.
Reproducibility
The paper is an arXiv preprint. While the methodology is described, the lack of a provided code repository or detailed hyperparameter settings in the abstract limits immediate reproducibility. However, the use of standard benchmarks (Adroit) and well-known techniques (diffusion policies, Q-learning) suggests that reproduction is feasible for skilled practitioners with access to the necessary simulation environments and hardware.
Limitations
The primary limitation is the reliance on a learned Q-function, which may require additional training or tuning for each new task, potentially offsetting the "test-time" nature of the steering. The anomaly score's effectiveness depends on the quality of the training data; if the training distribution is narrow, the anomaly signal might be overly conservative. Additionally, the computational cost of running a Q-function and anomaly detector alongside the generative policy at test time could be a bottleneck for real-time control, though this is not explicitly discussed in the abstract.
Broader impact
This work has significant implications for the robustness of generative robot policies. By providing a mechanism to prevent covariate shift without retraining, it offers a practical solution to a major challenge in deploying learned policies. The success in sim-to-real transfer suggests that this approach could be widely adopted in industrial robotics and other domains where safety and reliability are paramount. It bridges the gap between high-capacity generative models and the need for safe, constrained behavior. GUARD introduces an anomaly-informed Q-steering framework that enhances the robustness of generative robot policies by using dense anomaly scores to guide the denoising process, achieving significant improvements in success rates for dexterous manipulation tasks and demonstrating effective sim-to-real transfer. The paper presents a compelling solution to the covariate shift problem in closed-loop execution, offering a test-time steering mechanism that complements sparse rewards with failure-anticipatory signals, thereby improving both nominal and perturbed performance without modifying the pretrained policy.
Author of this paper? Send a correction.