WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation
Abstract
Most agentic robotic systems use vision-language models (VLMs) either to orchestrate predefined skills or to directly generate actions, constraints, and control programs. The former is limited by the available skill library, while the latter requires VLMs to reason about precise geometry and contact. In this paper, we propose WorldSketch, a new paradigm for agentic robotics in which a VLM constructs a world model rather than generating robot actions. The VLM acts as an agent that actively queries perception tools to build and visually validate a minimal kinematic world model, capturing only task-relevant objects, their interaction geometry, and possible motions. A geometric planner then jointly optimizes contact and whole-body motion on the resulting model. Moreover, the world model is self-improving: execution feedback verifies outcomes, updates uncertain hypotheses, and enables the agent to revise the model when its predictions are contradicted. We demonstrate the generality of this paradigm across RoboCasa, LIBERO-PRO, and R2R-CE navigation, achieving zero-shot success rates of 91%, 97%, and 76%, respectively. On RoboCasa, WorldSketch outperforms the next-best zero-shot agent baseline by 58 percentage points while using 10 times fewer VLM input tokens; on LIBERO-PRO, it exceeds CaP-X by 82 percentage points. In navigation, it matches the success rate of step-by-step VLM agents while using 6 times fewer input tokens.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel "WorldSketch" paradigm where VLMs construct minimal kinematic world models for robotic planning, achieving state-of-the-art zero-shot success rates in manipulation and navigation tasks while significantly reducing token usage compared to code-generation agents. By decoupling semantic scene understanding from geometric motion planning, the method demonstrates that explicit, verifiable world models built by VLMs can outperform end-to-end action generation, offering a scalable and interpretable approach to general-purpose robotics.
Methodology
The paper proposes "WorldSketch," a paradigm where a Vision-Language Model (VLM) constructs a minimal, kinematic world model (sketch) rather than generating actions directly. The VLM acts as an agent querying perception tools (depth lifting, point marking) to build a model of task-relevant entities (joints, links, parts). A geometric planner then optimizes contact and whole-body motion on this explicit model. The method includes a self-improving loop where execution feedback verifies outcomes and updates joint hypotheses via Bayes' rule. The methodology is sound, leveraging the VLM's semantic understanding for scene abstraction and the planner's precision for geometry. The separation of concerns (VLM for "what matters," Planner for "how to move") is a strong architectural choice that addresses the limitations of end-to-end VLA models and code-as-policy agents.
Results & evaluation
The evaluation is extensive, covering RoboCasa (kitchen manipulation), LIBERO-PRO (tabletop), and R2R-CE (navigation). The results are impressive, with 91% success on RoboCasa zero-shot, significantly outperforming baselines like CaP-X (33%) and Code as Policies (9%). The ablation studies are rigorous, isolating the impact of grounding methods (points vs. masks), overlay checks, and hypothesis testing. The comparison against trained VLAs (GR00T, pi0.5) demonstrates the superiority of the zero-shot agentic approach in generalization. The token efficiency analysis (10x fewer tokens than CaP-X) is a valuable practical insight.
Reproducibility
The paper provides detailed descriptions of the sketch language, tool interfaces, and planner components. However, the system relies on proprietary VLMs (GPT-6.1-Sol, GPT-6-Astra) and specific simulation environments (RoboCasa365, MuJoCo). The exact prompts and backend logic are described but not fully open-sourced in the text provided. The reliance on specific VLM versions may limit immediate reproducibility for those without access to these models, but the architectural principles are clearly defined.
Limitations
The system is evaluated primarily in simulation. Real-world sensor noise (depth errors >1cm) degrades performance. The current sketches cover only rigid parts with single joints, excluding deformable objects and complex multi-joint chains. The VLM latency (15s median) adds significant time to the loop. Contact-rich execution on small/thin parts remains a challenge, with failures often due to gripper limitations rather than planning errors.
Broader impact
This work offers a compelling alternative to the dominant VLA trend, suggesting that explicit world modeling by VLMs can achieve superior zero-shot generalization with lower computational cost. It bridges the gap between high-level semantic reasoning and low-level geometric planning. The paradigm of "sketching" the world is intuitive and could be extended to other robotic domains, potentially influencing how future agentic systems are designed to interact with physical environments. The paper introduces a novel "WorldSketch" paradigm where VLMs construct minimal kinematic world models for robotic planning, achieving state-of-the-art zero-shot success rates in manipulation and navigation tasks while significantly reducing token usage compared to code-generation agents. By decoupling semantic scene understanding from geometric motion planning, the method demonstrates that explicit, verifiable world models built by VLMs can outperform end-to-end action generation, offering a scalable and interpretable approach to general-purpose robotics.
Author of this paper? Send a correction.