World In-Context Learning for Agentic Robot Manipulation
Abstract
Trained action policies write manipulation skill into network weights, tying what is learned to the distribution it is learned from. In-context learning (ICL) avoids this coupling but has not matched trained policies, because a demonstration recorded in another scene carries no feedback from the world the robot currently faces. We observe that what an agent learns in context is semantic, recording task steps and object roles rather than visual appearance, so experience gathered in one world transfers to another. Based on this insight, we propose World In-Context Learning (WICL), a paradigm in which a frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes the feedback into its context rather than its weights. We build RoboScript, a modular infrastructure that implements WICL for agentic manipulation. From a single demonstration, RoboScript learns each task through rehearsal in simulation, freezes the learned knowledge as a Task Memory, and deploys the frozen memory on new scenes with data isolation that ensures every deployment input is available on a real robot. On all 50 tasks of RoboTwin 2.0, RoboScript reaches 83.4% mean success across clean and randomized settings with one demonstration per task and no gradient updates, exceeding the strongest trained action policy by over 20 points. The method generalizes across visual distributions, improves through reflection, scales with the foundation model, and transfers to a physical dual-arm robot through cross-embodiment memory adaptation. With a world in the loop, a robot can learn a manipulation task in its context as well as it learns one in its weights. All code is available at https://github.com/ArcheOne-AI/RoboScript
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces World In-Context Learning (WICL), a paradigm that enables robots to learn manipulation tasks by rehearsing in a simulator and storing semantic feedback in a frozen model's context, achieving state-of-the-art performance with a single demonstration. This represents a significant shift from weight-based policy learning to context-based agentic learning, offering superior generalization and data efficiency.
Methodology
The paper proposes World In-Context Learning (WICL), a paradigm that decouples task skill acquisition from weight updates by leveraging a "world" (simulator) to provide feedback into the context of a frozen foundation model. The core innovation is the "Task Memory," a structured Markdown document that stores semantic knowledge (object roles, relative geometry, workflow) rather than visual features or absolute coordinates. The methodology employs a rigorous data isolation protocol where the acting agent only sees deployable data (RGB-D, proprioception), while privileged simulator data is used only for supervision during the learning phase. The system uses a "memory linter" to enforce semantic abstraction, rejecting coordinates and ground-truth terms. The approach combines imitation (from a single demo) with rehearsal (in simulation) to refine this memory, followed by recursive self-improvement (RSI) where the agent reflects on its own failures to update the guidance.
Results & evaluation
The evaluation is extensive, covering all 50 tasks of the RoboTwin 2.0 benchmark. The method achieves 83.4% mean success with only one demonstration per task, significantly outperforming trained policies (which require 50 demos) by over 20 points, particularly under domain randomization. Ablation studies confirm that both the demonstration and the world feedback are necessary. The paper demonstrates generalization across visual distributions (retaining >95% success under randomization) and cross-embodiment transfer to a physical dual-arm robot without retraining. The cost analysis is transparent, showing the trade-off between model capability and API costs.
Reproducibility
The authors provide a GitHub repository with the code. The paper details the modular infrastructure, the specific foundation models used (GPT-6.1-Sol, etc.), and the protocol parameters. However, reliance on specific, potentially proprietary or rapidly changing LLM APIs (e.g., "GPT-6.1-Sol") may limit immediate reproducibility for the broader community if these models are not accessible. The strict data isolation and linter rules are well-defined, aiding in understanding the constraints.
Limitations
The method relies heavily on the capabilities of the underlying foundation model, which may not be available to all researchers. The cost of learning and deployment via API calls can be significant (up to $1,911 for full evaluation). The real-robot experiments are limited to four tasks, and the transfer relies on a lightweight adapter that may not generalize to all embodiment differences. The "world" is currently a simulator; applying this to real-world rehearsal without a simulator is not fully explored.
Broader impact
This work challenges the dominant paradigm of training VLA models from scratch for each task. By showing that semantic knowledge can be stored in context and refined via world feedback, it opens a path toward more flexible, data-efficient, and interpretable robot learning systems. The concept of "World In-Context Learning" could extend to other domains where simulation is available, such as autonomous driving or industrial automation. The paper introduces World In-Context Learning (WICL), a paradigm that enables robots to learn manipulation tasks by rehearsing in a simulator and storing semantic feedback in a frozen model's context, achieving state-of-the-art performance with a single demonstration. This represents a significant shift from weight-based policy learning to context-based agentic learning, offering superior generalization and data efficiency.
Author of this paper? Send a correction.