RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions
Abstract
Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces RynnVLA-Latent, a scalable framework for training VLA models by learning 3D-grounded latent actions from heterogeneous video data, significantly improving performance on robotic manipulation benchmarks by decoupling general motion learning from embodiment-specific control.
Methodology
The paper proposes RynnVLA-Latent, a framework addressing the data scarcity and embodiment mismatch problems in Vision-Language-Action (VLA) models. The core innovation is RynnLAM, a module that learns continuous, 3D-grounded latent actions from heterogeneous video data (human, robot, simulation) without requiring native robot control labels. The method employs residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling to ensure the latent space captures control-relevant motion rather than just visual appearance changes. The VLA model then uses flow matching to predict these latent action chunks from visual and language inputs, followed by an adaptation layer to map latent actions to specific robot embodiments. This approach effectively decouples the learning of general manipulation dynamics from the specific control spaces of individual robots, allowing for pretraining on massive, unstructured video datasets.
Results & evaluation
The experimental results are strong and comprehensive. The model achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus, 51.9% on VLABench, and 88.6% on RoboTwin 2.0. These numbers are competitive with or superior to state-of-the-art methods in the VLA domain. The evaluation includes both simulation benchmarks and real-world tasks across different robot platforms (dexterous hands, mobile grippers), demonstrating generalization. The use of 30.1K hours of video data to generate 745.52M latent-action descriptors highlights the scalability of the approach. Ablation studies on representation learning and data scaling further support the technical claims.
Reproducibility
The paper provides detailed descriptions of the RynnLAM training process and the flow matching objective. However, specific hyperparameters, exact dataset compositions, and code availability are not explicitly provided in the extracted text. The reliance on a large-scale proprietary or curated dataset (30.1K hours) may limit immediate reproducibility for smaller labs, though the methodology is clearly defined.
Limitations
The primary limitation is the computational cost of processing 30.1K hours of video and generating latent actions. Additionally, the mapping from latent actions to specific robot actions requires an adaptation phase, which may not generalize perfectly to entirely new embodiments without some fine-tuning. The performance on VLABench (51.9%) is lower than on other benchmarks, suggesting potential weaknesses in certain complex or long-horizon tasks.
Broader impact
This work has significant implications for the field of robotic learning by providing a scalable pathway to train foundation models using unstructured video data. It reduces the dependency on expensive, robot-specific action-labeled datasets, potentially accelerating the development of general-purpose robot policies. The 3D-grounded latent action space offers a robust representation that can be shared across different robot morphologies. The paper introduces RynnVLA-Latent, a scalable framework for training VLA models by learning 3D-grounded latent actions from heterogeneous video data, significantly improving performance on robotic manipulation benchmarks by decoupling general motion learning from embodiment-specific control.
Author of this paper? Send a correction.