MotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding
Abstract
Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense, the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with egocentric video and yields synchronized video, bimanual EMG and IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producing an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces MotorSense, a large-scale paired egocentric video and EMG dataset, and demonstrates that cross-modal sensorimotor alignment significantly enhances action-centric computer vision tasks. By leveraging EMG as a privileged supervisory signal in a CLIP-style pre-training framework, the authors achieve consistent state-of-the-art results in action recognition, 3D hand reconstruction, and contact prediction, while also enabling fine-grained control of video diffusion models through learned motor codes.
Methodology
The paper introduces a novel cross-modal pre-training framework that aligns egocentric video with surface electromyography (EMG) signals. The core methodological contribution is the use of EMG as a "privileged supervisory signal" in a CLIP-style contrastive objective. By jointly training a video encoder and an EMG encoder, the authors create a shared embedding space where visual representations are shaped by the underlying motor intent, which is then discarded at inference time. This approach is technically sound, leveraging the dense, occlusion-free nature of EMG to provide a stronger inductive bias for action-centric vision tasks than standard self-supervised video learning. The inclusion of a VQ-VAE to learn discrete "motor codes" for conditioning video diffusion models is a creative extension, though the reliance on the Video2LoRA framework limits the novelty of the generative component itself.
Results & evaluation
The experimental evaluation is extensive and rigorous. The authors demonstrate consistent improvements over strong baselines (SimCLR, MoCo, BYOL, and ImageNet/MAE initialization) across four distinct task families: action recognition (EPIC-Kitchens, SSv2, Ego-Exo4D), 3D hand reconstruction (ARCTIC, H2O, HOT3D), 3D contact reconstruction, and video generation. The gains are particularly notable in the frozen-backbone setting for action recognition, suggesting the pre-training captures robust, transferable features. The comparison against RGB-only self-supervised methods trained on the same data effectively isolates the contribution of the EMG signal. The qualitative results for video diffusion show clear improvements in fine-grained hand-object interaction fidelity compared to language or video-history conditioning.
Reproducibility
The paper provides high reproducibility. It details the specific hardware (MindRove Armband, Meta Aria glasses), data collection protocols, preprocessing steps (filtering, normalization), and model architectures. The release of the 100-hour dataset is a significant asset. However, the specific implementation of the "Video2LoRA" and "HyperNetwork" components relies on external citations, and the code for the VQ-VAE motor code discovery is described but not fully detailed in the main text, potentially requiring access to the supplementary materials or code repository (not explicitly linked in the provided text, though a project page is mentioned) for full replication.
Limitations
The primary limitation is the scale of the dataset (100 hours) compared to massive video datasets like Kinetics or Ego4D. The EMG hardware used is consumer-grade (8 channels), which may limit the granularity of muscle activity captured compared to research-grade high-density EMG. The video diffusion experiments are limited to a specific backbone (CogVideoX-Fun) and a lightweight LoRA approach, which may not generalize to all state-of-the-art video generation models. Additionally, the dataset is focused on kitchen/household manipulation, which may limit generalizability to other domains.
Broader impact
This work has significant potential impact on embodied AI, robotics, and VR/AR. By providing a dataset and method that links visual perception with motor intent, it offers a pathway to more intuitive human-robot interaction and better understanding of human actions from video. The motor-code conditioning for video generation could enable new tools for content creation and simulation. The dataset itself becomes a valuable resource for the community, addressing a gap in paired video-EMG data. The paper introduces MotorSense, a large-scale paired egocentric video and EMG dataset, and demonstrates that cross-modal sensorimotor alignment significantly enhances action-centric computer vision tasks. By leveraging EMG as a privileged supervisory signal in a CLIP-style pre-training framework, the authors achieve consistent state-of-the-art results in action recognition, 3D hand reconstruction, and contact prediction, while also enabling fine-grained control of video diffusion models through learned motor codes.
Author of this paper? Send a correction.