Machine Learning PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-15

MoViC: Real-Time Humanoid Whole-Body Control from Monocular Video Streams

Yimin Han, Jiahui Zhang, Wanyue Li, Zeren Luo, Yeke Chen, Liu Zhao, Peizhuo Li, Xiaoyu Ji, Shihao Dong, Xuechao Chen, Zhangguo Yu, Annan Tang, Peng Lu

The University of Hong Kong +3 · arXiv

Affiliations: The University of Hong Kong, National University of Singapore, Beijing Institute of Technology, EngineAI

Abstract

Real-time whole-body imitation from monocular video requires both a robust universal tracker and high-quality real-time reconstruction. We present MoViC, a framework that uses live RGB video as a control signal for humanoid robots without wearable tracking devices. The framework combines monocular human motion reconstruction, robot retargeting, and a transformer-based behavior foundation model. The policy receives proprioceptive history, a single explicit next-step reference, and a compact frequency-domain motion representation that supplies temporal context without explicitly stacking future reference frames, which reduces the input dimension and network size. During deployment, our extrapolation method bridges the 10-Hz visual estimation with 50-Hz controller, enabling asynchronous visual processing and policy execution. Hardware experiments further demonstrate highly dynamic motion tracking and device-free teleoperation of bimanual and single-arm manipulation tasks from live monocular RGB input. Project page: https://movicproject.github.io/ .

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

MoViC enables real-time humanoid whole-body control from monocular video by integrating high-quality visual reconstruction with a frequency-domain conditioned transformer policy and latent-dynamics extrapolation. The paper presents a robust and practical system that bridges the gap between slow visual estimation and fast control requirements, demonstrating significant improvements in tracking dynamic motions without wearable sensors.

Methodology

The paper proposes MoViC, a framework for real-time humanoid whole-body control (WBC) driven by monocular RGB video. The core technical contributions are threefold: (1) A "SEG" module that integrates SAM 3D Body for per-frame human reconstruction at ~10 Hz, followed by retargeting and a latent-dynamics extrapolation method to generate 50 Hz reference motions. This addresses the critical latency mismatch between visual estimation and control rates. (2) A transformer-based behavior foundation model (BFM) that uses a compact frequency-domain motion representation (DFM) instead of stacking future reference frames. This reduces input dimensionality and network size while providing temporal context. (3) An asymmetric actor-critic training setup using PPO, where the actor receives proprioceptive history, a single next-step reference, and the frequency-domain goal window. The methodology is sound, leveraging existing strong components (SAM 3D Body, DFM) but integrating them into a novel real-time pipeline with specific architectural choices for efficiency and robustness.

Results & evaluation

The evaluation is comprehensive, covering simulation (MuJoCo) and real-robot experiments on the EngineAI PM01 platform. The authors compare their video reconstruction module against state-of-the-art methods (GVHMR, HybrIK) on standard datasets (EMDB) and custom dynamic dance sequences, showing superior performance in MPJPE and PCK. For tracking, they compare against SONIC, OmniH2O, and TWIST, demonstrating lower joint position and velocity errors on unseen dynamic motions. The real-robot experiments validate the system's ability to perform bimanual and single-arm manipulation tasks without wearable devices. The use of a specific dance ("Bencaogangmu") as a stress test for dynamic motion is a strong experimental choice.

Reproducibility

The paper provides detailed implementation details, including hardware specs (RTX 5090, Logitech BRIO), software libraries (MuJoCo, TensorRT), and hyperparameters for training. The use of open-source components like SAM 3D Body and standard RL algorithms (PPO) aids reproducibility. However, the specific retargeting optimization and the proprietary nature of the EngineAI PM01 robot may limit full reproduction by other groups without access to similar hardware. The project page is provided, which likely contains code or further details.

Limitations

The system relies on a fixed camera setup with calibrated intrinsics and gravity direction, limiting its applicability to mobile or uncalibrated camera scenarios. The visual reconstruction runs at ~10 Hz, which, while bridged by extrapolation, still introduces potential lag or error accumulation in highly unpredictable motions. The paper focuses on a single robot platform (EngineAI PM01), and generalization to other humanoid morphologies is not extensively tested. The frequency-domain representation, while efficient, may struggle with highly non-periodic or chaotic motions if the local reconstruction assumption fails.

Broader impact

This work significantly advances the field of vision-based humanoid teleoperation by removing the need for expensive MoCap or VR hardware. It demonstrates that high-quality, real-time whole-body control is feasible with a simple RGB camera, making humanoid robots more accessible for research and deployment in human-centric environments. The integration of frequency-domain representations for policy conditioning offers a scalable approach for handling large motion datasets, which could influence future designs of behavior foundation models. MoViC enables real-time humanoid whole-body control from monocular video by integrating high-quality visual reconstruction with a frequency-domain conditioned transformer policy and latent-dynamics extrapolation. The paper presents a robust and practical system that bridges the gap between slow visual estimation and fast control requirements, demonstrating significant improvements in tracking dynamic motions without wearable sensors.

Author of this paper? Send a correction.