WOVEN: Weaving Visual World Modeling into Multimodal LLMs
Abstract
Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect a shared deficit in visual transition reasoning, and test whether this capability can serve as a shared training primitive, one that different models can learn from different supervision sources and reuse across different tasks, with a systematic training recipe. Existing benchmarks document these deficits separately but do not support controlled comparisons across scenes, actions, and reasoning operations. We therefore introduce WOVEN, a training source and benchmark for visual transition reasoning that organizes transition supervision by scene, action, and reasoning type, using diverse, realistic rollouts from video-pretrained generative models: 36,076 examples across 20 scene types, 5 action types, and 8 reasoning types. We first evaluate 38 frontier MLLMs (e.g., GPT-5.4 and Qwen3-VL-235B-A22B) and find a substantial and systematic deficit: even the strongest models fall far below humans, and the failures recur across model families and persist with scale. We then train MLLMs at multiple scales on WOVEN and find that they learn a shared capability that transfers broadly: training subsets of only about 2,000 items each collectively improve 22 of 26 external benchmarks by up to 27.3 percentage points, and WOVEN data can replace 30-50% of a task's own training data with comparable accuracy. Controlled comparisons further yield a training recipe for visual world modeling, validated prospectively on held-out benchmarks: select supervision by the reasoning operation it teaches rather than by the actions, scenes, or domains it shows, and prefer larger changes to the visual state for robustness. Our work establishes visual transition reasoning as a reusable foundation for systematic visual world-model training in MLLMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper establishes visual transition reasoning as a shared, trainable primitive for multimodal LLMs and provides a systematic training recipe validated by a new benchmark, WOVEN. By demonstrating that small, targeted subsets of transition supervision can broadly improve performance across diverse spatial, physical, and temporal benchmarks, the work offers a data-efficient and theoretically grounded approach to enhancing visual world modeling capabilities in current MLLMs.
Methodology
The paper proposes a novel framework for "visual transition reasoning," formalizing it as inference over $(s, a, s')$ triplets. The core methodological contribution is the WOVEN dataset, which uses video-pretrained generative models (Wan2.2) to create controlled rollouts with known actions and endpoints. This allows for the construction of multiple-choice questions with error-typed distractors, enabling fine-grained analysis of model failures. The training methodology involves supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) on controlled subsets of the data. The systematic ablation study comparing supervision by reasoning operation versus action type is a strong methodological choice that isolates the source of transfer.
Results & evaluation
The experimental evaluation is extensive and rigorous. The authors evaluate 38 frontier MLLMs to establish a baseline deficit. They then demonstrate that training on small subsets (~2,000 items) of WOVEN significantly improves performance on 22 of 26 external benchmarks. A key strength is the causal intervention experiment, where replacing 30-50% of task-specific training data with WOVEN data yields comparable accuracy, proving the utility of the dataset as a data-efficient substitute. The correlation analysis ($r=0.86$) between WOVEN accuracy gains and external benchmark gains provides strong evidence for the "shared primitive" hypothesis.
Reproducibility
The paper provides high reproducibility. Code and datasets are released on GitHub and HuggingFace. The generation process using Wan2.2 is specified, and the training recipes (SFT + GRPO) are detailed. The use of standard benchmarks for evaluation ensures that results can be verified against existing baselines.
Limitations
The reliance on synthetic video generation (Wan2.2) may introduce domain gaps compared to real-world physics, although the authors argue for realism. The evaluation is primarily on multiple-choice tasks, which may not fully capture open-ended reasoning capabilities. The "frontier" models evaluated include some very recent releases (GPT-5.4, Qwen3-VL), which may limit immediate reproducibility for labs without access to these specific API versions or weights.
Broader impact
This paper has significant potential to influence how multimodal models are trained for embodied AI and physical reasoning. By identifying visual transition reasoning as a reusable primitive, it offers a data-efficient path to improving spatial and temporal understanding. The systematic recipe for selecting supervision (by reasoning operation rather than action) provides actionable guidelines for practitioners building world models. The paper establishes visual transition reasoning as a shared, trainable primitive for multimodal LLMs and provides a systematic training recipe validated by a new benchmark, WOVEN. By demonstrating that small, targeted subsets of transition supervision can broadly improve performance across diverse spatial, physical, and temporal benchmarks, the work offers a data-efficient and theoretically grounded approach to enhancing visual world modeling capabilities in current MLLMs.
Author of this paper? Send a correction.