Perceptive Humanoid Motion from Simulation
Abstract
Humanoid robots have the potential to automate a wide range of physical tasks. Progress toward generalist humanoid autonomy, however, remains constrained by a data bottleneck: mapping pixels and free-form language to low-level joint commands requires paired supervision, and no corpus of such data exists at sufficiently large scale. Some approaches rely on human teleoperation and wearable capture devices, which require at least an hour of human time to produce an hour of data. We present HALO, a humanoid autonomy system powered by a pipeline that synthesizes vision-language-motion data across a thousand procedurally generated homes. The corpus scales with compute rather than human labor. HALO acquires a repertoire of physical skills, including sitting on chairs, navigating to named objects, stepping sideways through half-open doors, crouching and crawling beneath tables, and pushing boxes. It orchestrates these skills to solve composite, long-horizon tasks. Without ever seeing a real home during training, HALO transfers zero-shot to the physical world, achieving a 91% success rate across 360 trials in six unmapped real-world environments. See https://apple-aiml-research.github.io/ml-halo/ for code and videos.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
HALO demonstrates that simulation-based data synthesis, combined with a hierarchical VLM-diffusion-RL architecture, enables zero-shot transfer of perceptive humanoid skills to unseen real-world environments. The paper makes a significant contribution by decoupling data generation from human labor, providing a scalable and rigorous framework for training generalist humanoid autonomy systems.
Methodology
The paper introduces HALO, a hierarchical autonomy stack for humanoid robots that decouples semantic reasoning (VLM orchestrator), motion planning (diffusion-based generator), and physical execution (RL-based tracker). The core methodological innovation lies in the data generation pipeline, which synthesizes 300 hours of vision-language-motion data using procedural scene generation (Infinigen) and kinematic motion diffusion (Kimodo), entirely bypassing human teleoperation. The use of a frozen VLM encoder paired with a DiT decoder for motion generation is a strong architectural choice, leveraging web-scale semantic alignment. The asynchronous generation strategy, which conditions the diffusion model on the first 0.5s of the trajectory to ensure continuity without inference-time patching, is a clever engineering solution to the latency-stability trade-off in real-time control. The tracker training via multi-teacher distillation followed by multi-critic PPO fine-tuning with depth inputs is rigorous and addresses the sim-to-real gap effectively.
Results & evaluation
The evaluation is robust, featuring 360 real-world trials across six unmapped environments, achieving a 91% success rate for single tasks. The ablation studies are particularly valuable, isolating the impact of environment diversity (showing a sharp drop in performance with fewer than 100 environments), the necessity of the diffusion decoder over regression, and the critical role of depth perception in the tracker. The multi-task evaluation reveals realistic limitations in long-horizon chaining, providing honest insight into current system boundaries. The scale of synthetic data (1,000 homes) and the compute resources used (250 H100s for data gen) are clearly stated, adding credibility to the results.
Reproducibility
The paper provides high-level architectural details and references to specific components (Qwen3.5, Kimodo, Infinigen). However, as is common with industry papers, the exact code for the proprietary pipeline components (HALO-specific orchestrator logic, custom reward functions) may not be fully open-sourced. The availability of a project page with videos and code links helps, but the reliance on specific commercial hardware (Unitree G1, Mac mini, Jetson Orin) and proprietary VLMs may limit immediate reproducibility for academic labs without similar resources.
Limitations
The system requires human-authored keyframes and prompts for new tasks, limiting the speed of repertoire expansion. The reliance on vision for contact perception is a significant weakness for manipulation tasks involving occlusion. The orchestration layer is described as relatively simple, potentially struggling with highly complex or ambiguous long-horizon tasks. The compute cost for data generation, while decoupled from human labor, remains substantial (48 hours on 250 H100s).
Broader impact
This work significantly advances the field of humanoid robotics by demonstrating that large-scale, diverse, and physically plausible training data can be synthesized without human intervention. It challenges the prevailing paradigm of teleoperation-based data collection and provides a scalable pathway toward generalist humanoid autonomy. The zero-shot transfer to unseen real-world environments is a major milestone, suggesting that simulation-based training can effectively bridge the sim-to-real gap for complex, perceptive tasks. This approach could be adapted to other embodied AI domains, enabling faster development of autonomous agents in unstructured environments. HALO demonstrates that simulation-based data synthesis, combined with a hierarchical VLM-diffusion-RL architecture, enables zero-shot transfer of perceptive humanoid skills to unseen real-world environments. The paper makes a significant contribution by decoupling data generation from human labor, providing a scalable and rigorous framework for training generalist humanoid autonomy systems.
Author of this paper? Send a correction.