Duo: A Reciprocal World Model for Full-Duplex Interaction
Abstract
In this report, we introduce the Reciprocal World Model (RWM) and present Duo v1, a full-duplex world model that continuously perceives audiovisual inputs while responding to users in real time. The RWM treats intelligence and physical reality as equally fundamental to what constitutes a world. Existing world models largely focus on simulating the latter, leaving reasoning and decision-making as external controls rather than intrinsic parts of the world itself. In RWM, we identify two equally fundamental capabilities: Intent, which provides the cognition underlying world interactions, and Realization, which generates coherent audio-visual behavior. To realize this formulation, Duo v1 introduces three key technical designs: (1) Leveraging autoregressive modeling for continuous interaction reasoning and participation decisions, rather than restricting it to instruction understanding for diffusion models. (2) Constructing chunk-specific conditions and synchronize tokens across Intent and Realization on a unified timeline. Module-specific attention masks enforce causal dependencies both within and across chunks, preventing access to future inputs. (3) Mitigating distribution shift in both modules through dual rollout distillation, training on autoregressively generated Intent tokens and synthesized audiovisual histories from Realization. Both modules continuously perceive real-time audio, video, and text, enabling full-duplex interaction with spontaneous conversations and interruptions. Beyond conversation, users can share real objects, reshape virtual scenes, guide assistants, or play audiovisual games without mouse or keyboard controls. The release of Duo v1 makes our first step, with the next version coming soon. We look forward to broader exploration of this emerging field.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces the Reciprocal World Model (RWM) and Duo v1, a full-duplex interaction system that unifies autoregressive reasoning and audiovisual generation on a synchronized timeline. The technical contribution is significant in its architectural approach to real-time multimodal interaction, but the lack of detailed experimental evidence in the provided text limits the assessment of its practical impact and robustness.
Methodology
The paper proposes the Reciprocal World Model (RWM) framework, implemented as Duo v1, which integrates two distinct capabilities: Intent (cognition/reasoning) and Realization (audio-visual generation). The core technical contribution is the unification of autoregressive modeling for continuous interaction reasoning with diffusion-based generation for audiovisual output. The authors introduce a unified timeline with chunk-specific conditions and synchronize tokens to align the Intent and Realization modules. A key architectural feature is the use of module-specific attention masks to enforce causal dependencies, preventing the model from accessing future inputs during real-time processing. To address the distribution shift inherent in autoregressive generation, the paper employs dual rollout distillation, training the models on their own autoregressively generated tokens and synthesized histories. This approach allows for full-duplex interaction, where the system can perceive and respond simultaneously, handling interruptions and spontaneous conversations.
Results & evaluation
The provided text is a truncated report containing only section headers and the abstract. It lacks detailed experimental results, quantitative benchmarks, or comparative analysis against baseline models. While the abstract claims capabilities such as sharing real objects, reshaping virtual scenes, and playing games without mouse/keyboard, there is no empirical evidence in the provided text to verify the quality, latency, or robustness of these interactions. The evaluation section is referenced but not included in the text.
Reproducibility
Reproducibility is currently low due to the lack of detailed implementation specifics, hyperparameters, dataset descriptions, and code availability in the provided text. The mention of "Duo v1" suggests a specific release, but without access to the full technical report or code repository, it is difficult to assess the ease of reproduction. The complexity of the dual-module architecture and the specific attention masking strategies would require significant engineering effort to replicate.
Limitations
The primary limitation in this evaluation is the incomplete nature of the provided text, which prevents a rigorous assessment of experimental validity. The paper is described as a "report" and "first step," implying that the system may be in an early stage of development. The reliance on autoregressive modeling for real-time interaction may introduce latency challenges, although the authors claim to mitigate this with chunking and synchronization. The distribution shift issue, while addressed by distillation, remains a fundamental challenge in autoregressive multimodal models.
Broader impact
The concept of a "Reciprocal World Model" that treats intelligence and physical reality as equally fundamental has significant implications for the field of embodied AI and human-computer interaction. If successful, this approach could lead to more natural and immersive interactive agents that can engage in complex, real-time dialogues and actions. The ability to interact with virtual scenes and objects without traditional input devices could revolutionize gaming, virtual reality, and assistive technologies. However, the impact is currently speculative pending rigorous evaluation and broader adoption. The paper introduces the Reciprocal World Model (RWM) and Duo v1, a full-duplex interaction system that unifies autoregressive reasoning and audiovisual generation on a synchronized timeline. The technical contribution is significant in its architectural approach to real-time multimodal interaction, but the lack of detailed experimental evidence in the provided text limits the assessment of its practical impact and robustness.
Author of this paper? Send a correction.