On Agentic Manipulation Policies
Abstract
Recent work on robot manipulation has focused on specialist policies trained on demonstrations. We show that generalist AI models can control manipulators more reliably than specialist policies such as vision-language-action models (VLAs) and world-action models (WAMs). We systematically analyze agentic manipulation policies (AMPs), in which a generalist AI model commands the manipulator directly, adapting its actions as the task unfolds. We present AMP1, a reference harness that runs unchanged across single-arm, bimanual, and mobile manipulation. AMP1 achieves state-of-the-art results on the RoboLab-120 and CaP-Gym benchmarks, with no demonstrations or task-specific training. On RoboLab-120, AMP1 reaches an 85.6\% success rate zero-shot, compared to 49.2\% and 42.9\% for leading VLA and WAM policies. The agent succeeds zero-shot in 104 of 108 real-world trials spanning twelve tasks and three robot instances, reducing failure rates by 20x and 15x relative to strong VLA and WAM baselines. The basic harness supplies no grasping, pushing, or placing skills; the agent develops its own strategies, such as searching for hidden objects, using objects as tools, and calibrating its cameras. With the addition of a persistent memory store, the agent can also learn from experience: an augmented harness with a memory representation that is carried across episodes lowers the failure rate on RoboDojo bimanual manipulation tasks by 4.6x relative to zero-shot deployment. Our analysis suggests that AMPs are substantially more general and robust than specialist manipulation policies, providing a promising foundation for robot manipulation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper demonstrates that generalist AI models, when given direct control interfaces and persistent memory, can outperform specialist vision-language-action models in robot manipulation tasks. By systematically analyzing the interplay between model capability, reasoning effort, and harness design, the authors show that agentic policies enable emergent, robust behaviors like tool use and self-calibration, establishing a new foundation for generalist robot control.
Methodology
The paper proposes "Agentic Manipulation Policies" (AMPs), specifically the AMP1 harness, which allows a generalist Large Language Model (LLM) to directly control a robot manipulator via code generation and direct motion commands, bypassing the need for specialist Vision-Language-Action (VLA) models or pre-defined skill libraries. The methodology is rigorous, involving a systematic ablation of model choice, reasoning effort, and sensory inputs (RGB, Depth, Calibration). A key methodological contribution is the demonstration that stronger models perform *worse* when given assistive high-level skills (grasping/placing routines) because these routines hide failure modes that the model could otherwise detect and correct. The paper also introduces a gradient-free learning mechanism where the agent maintains a persistent memory store of code and notes to improve performance across episodes without weight updates.
Results & evaluation
The experimental evaluation is extensive and compelling. The authors test on four simulation benchmarks (RoboLab-120, LIBERO-PRO, robosuite, BEHAVIOR-1K) and real-world hardware (Franka Research 3). The results show a massive improvement over state-of-the-art specialist policies (VLAs and WAMs), achieving 85.6% success on RoboLab-120 zero-shot compared to ~50% for baselines. The real-world trials (104/108 success) are particularly strong, demonstrating emergent behaviors like tool use, self-calibration, and non-prehensile manipulation. The ablation studies are thorough, isolating the impact of model capability, reasoning depth, and sensory degradation.
Reproducibility
The paper describes the harness architecture in detail, including the prompt structure, code execution environment, and motion planning interfaces. However, as is common with frontier model-based systems, the exact prompts and specific internal workings of the "frontier models" (Gemini 4 Argon, Claude Opus 5.5) are proprietary. The open-source models tested (Qwen, GLM, etc.) allow for some reproducibility, but the primary results rely on closed-source APIs. The code for the harness and evaluation scripts is not explicitly linked in the provided text, though the methodology is described with sufficient detail for replication in principle.
Limitations
The primary limitation is latency; the robot spends significant time waiting for the LLM to reason, making it unsuitable for high-speed or dynamic tasks. The paper acknowledges that the model cannot react in real-time to trajectory deviations during execution. Additionally, the approach relies heavily on the capabilities of the underlying LLM, meaning performance is tied to the progress of general AI models rather than specific robotics research. The cost of running such high-reasoning LLMs for every action step is also a practical barrier.
Broader impact
This paper has high potential impact by shifting the paradigm of robot manipulation from training specialized end-to-end policies to leveraging the general reasoning capabilities of LLMs. It suggests that as LLMs improve, their utility in robotics will grow without needing to retrain robot-specific models. The finding that direct control outperforms skill-based orchestration for strong models is a significant insight for the field. It also highlights the importance of "active perception" and self-correction in embodied agents. The paper demonstrates that generalist AI models, when given direct control interfaces and persistent memory, can outperform specialist vision-language-action models in robot manipulation tasks. By systematically analyzing the interplay between model capability, reasoning effort, and harness design, the authors show that agentic policies enable emergent, robust behaviors like tool use and self-calibration, establishing a new foundation for generalist robot control.
Author of this paper? Send a correction.