Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration
Abstract
Human video scales easily and does not need robot hardware, but existing methods require depth sensing, overlay the robot onto 2D video, or generate data only for the demonstrated objects. We present Real2Sem2Real (R2Sem2R), a real-to-semantic-to-real pipeline that turns one monocular RGB video of a human demonstration into thousands of robot demonstrations across the semantic object instances of a category. R2Sem2R tracks the SE(3) pose of each object with DINOv3-distilled Gaussian splats, outperforming existing methods and FoundationPose with estimated metric depth. R2Sem2R then aligns digital cousins, each generated from one or a few photographs, to the demonstrated objects with a semantic alignment. This alignment matches whole objects and the functional regions of target objects, such as a coffee machine's tray. R2Sem2R maps the human hand to a feasible robot grasp that extends to the cousins. Robot demonstrations are generated either in IsaacSim, for photorealistic rendering, or MuJoCo, for contact physics, and R2Sem2R can generate over 2,000 episodes in about two hours. On two real robots and four tasks, we train a vision-language-action policy on data from one human video per task, and policies trained on our data match or exceed policies trained on teleoperation across diverse object instances. The project page is available at https://real2sem2real.github.io/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel and effective pipeline for generating semantically augmented robot demonstrations from a single human RGB video. By combining robust RGB-only object tracking with semantic alignment of digital cousins, it enables the transfer of functional knowledge across object instances, resulting in policies that match or exceed those trained on teleoperation with significantly less human effort.
Methodology
The paper proposes a coherent "Real-to-Semantic-to-Real" pipeline that addresses a critical bottleneck in robot learning: the cost of collecting diverse, object-specific demonstrations. The core technical contribution is a robust RGB-only 4D object tracker using DINOv3-distilled Gaussian splats, which outperforms depth-dependent baselines like FoundationPose. The novelty lies in the "semantic alignment" of digital cousins (generated from few photos) to the demonstrated objects, allowing the transfer of functional affordances (e.g., where to place a mug on a specific machine) rather than just geometric similarity. The integration of hand-to-gripper retargeting and parallel simulation in IsaacSim/MuJoCo is well-engineered, though the reliance on VLMs for keyframe selection and region segmentation introduces potential fragility.
Results & evaluation
The experimental setup is rigorous, involving two real robots (xArm7, Panda) and four distinct tasks that test different aspects of the pipeline (placement, pouring, hanging, reorientation). The comparison against teleoperation is the gold standard, and the results showing R2Sem2R matching or exceeding teleoperation with significantly less human effort are compelling. The ablation study isolating the effect of semantic augmentation (R2Sem2R vs. R2R2R+) is particularly valuable, clearly demonstrating the added value of the cousin alignment. The tracking accuracy benchmarks on YCBInEOAT provide strong evidence for the superiority of the proposed tracker.
Reproducibility
The paper provides detailed descriptions of the pipeline components, including loss functions, initialization strategies, and simulation settings. The use of standard tools (IsaacSim, MuJoCo, DINOv3) aids reproducibility. However, the specific implementation details of the "semantic ICP" and the VLM prompts used for region selection are not fully detailed in the main text, which could hinder exact replication. The code availability is implied by the project page, but the paper itself is a preprint/submission, so full code release status is pending.
Limitations
The method is limited to rigid objects and requires a metric measurement of the demonstrated objects, which is a non-trivial assumption. The IsaacSim kinematic replay lacks contact dynamics, limiting applicability to contact-rich tasks, though MuJoCo is used as an alternative. The pipeline is computationally intensive (2.5 hours for 2000 episodes) and requires careful setup of the scene and camera calibration. The reliance on monocular depth estimation for some components (though mitigated by the splat tracker) remains a potential source of error.
Broader impact
This work has significant potential to democratize robot learning by reducing the barrier to entry for data collection. By enabling the generation of thousands of diverse demonstrations from a single human video, it could accelerate the development of general-purpose robot policies. The approach is applicable to a wide range of manipulation tasks and could be extended to other embodiments and object categories. The paper presents a novel and effective pipeline for generating semantically augmented robot demonstrations from a single human RGB video. By combining robust RGB-only object tracking with semantic alignment of digital cousins, it enables the transfer of functional knowledge across object instances, resulting in policies that match or exceed those trained on teleoperation with significantly less human effort.
Author of this paper? Send a correction.