Interactive World Simulation with Cosmos for Physical AI
Abstract
We present a shared framework that adapts Cosmos world foundation models for real-time interactive world simulation in Physical AI. By streaming visual observations in response to actions and scene controls, the framework enables users and embodied agents to interact with evolving virtual environments. It supports both predicting the visual consequences of actions and generating observations from physical state maintained by an external simulator. We demonstrate this approach across camera navigation and robot manipulation, controllable video transfer, and multi-view autonomous vehicle (AV) simulation. These examples provide a shared foundation for developing interactive virtual environments for Physical AI training and evaluation. Our code and model checkpoints are available at https://github.com/nvidia/cosmos and https://huggingface.co/collections/nvidia/cosmos3-examples. The project website is available at https://research.nvidia.com/labs/cosmos-lab/cosmos-simulation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a shared framework for adapting Cosmos world foundation models to real-time interactive simulation, enabling physical AI agents to interact with evolving virtual environments. While the provided text is limited to structural elements, the combination of a strong institutional pedigree (NVIDIA), a novel dual-mode simulation framework, and full open-sourcing of code and weights suggests a significant contribution to the field of generative world models for robotics and autonomous systems.
Methodology
The paper proposes a framework for adapting Cosmos world foundation models to support real-time interactive simulation. The core method involves streaming visual observations in response to actions and scene controls, effectively bridging the gap between static video generation and dynamic, stateful simulation. It supports two distinct modes: predicting visual consequences of actions (action-conditioned generation) and generating observations from physical states maintained by an external simulator (state-conditioned generation). This dual-mode approach is methodologically sound for Physical AI, as it allows for both closed-loop agent interaction and integration with high-fidelity physics engines.
Results & evaluation
The evaluation demonstrates the framework across three key domains: camera navigation, robot manipulation (bimanual), and multi-view autonomous vehicle simulation. The inclusion of robot manipulation and AV simulation is significant, as these are high-stakes applications for world models. However, the provided text snippet is largely structural (LaTeX preamble and table of contents), lacking specific quantitative metrics (e.g., FID, FVD, success rates, latency benchmarks) or comparative baselines against other world models like Genie or Sora. The "Abstract-Only Score" of 60 suggests the abstract is strong, but the full text analysis is limited by the provided excerpt's lack of detailed results tables.
Reproducibility
High. The paper explicitly provides links to open-source code (GitHub) and open-weight model checkpoints (Hugging Face) for multiple variants (Bimanual, Transfer, Auto). This level of open-sourcing is a major strength, significantly enhancing reproducibility compared to many industry papers that only release code or only release weights.
Limitations
The provided text is primarily the LaTeX structure and front matter, lacking the detailed experimental results, ablation studies, and failure case analyses that would be present in the full body of the paper. Without access to the specific performance metrics and comparisons, it is difficult to rigorously assess the technical superiority over existing baselines. The reliance on "Cosmos" as a pre-existing foundation model means the novelty lies in the adaptation framework rather than the base model itself.
Broader impact
High potential. Interactive world models are a critical component for training embodied AI agents. By providing a shared, open framework for real-time simulation, this work could accelerate research in robotics and autonomous driving by providing a scalable, photorealistic training environment. The open-source nature further amplifies this impact by lowering the barrier to entry for the broader research community. The paper presents a shared framework for adapting Cosmos world foundation models to real-time interactive simulation, enabling physical AI agents to interact with evolving virtual environments. While the provided text is limited to structural elements, the combination of a strong institutional pedigree (NVIDIA), a novel dual-mode simulation framework, and full open-sourcing of code and weights suggests a significant contribution to the field of generative world models for robotics and autonomous systems.
Author of this paper? Send a correction.