RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Abstract
We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
RT-2 introduces a unified tokenization and co-fine-tuning recipe that transforms large vision-language models into generalist robotic policies capable of zero-shot semantic reasoning and cross-domain generalization. By demonstrating that web-scale pretraining can be directly leveraged for low-level control without architectural modifications, the paper fundamentally shifts the robotics community's approach to data scaling, establishing VLA models as a dominant paradigm for embodied AI while clearly delineating current boundaries in skill acquisition, inference latency, and compute accessibility.
Methodology
The core methodological contribution is elegantly minimal yet highly effective: discretizing continuous 6-DoF end-effector and gripper actions into 256 text tokens and injecting them directly into the output vocabulary of pre-trained vision-language models (PaLI-X, PaLM-E). By co-fine-tuning on both web-scale vision-language corpora and robotic trajectory data, the approach prevents catastrophic forgetting of semantic knowledge while grounding the model in physical control. The output constraint mechanism ensures valid action sampling during inference, and the chain-of-thought adaptation demonstrates a seamless bridge between high-level reasoning and low-level actuation. While the architectural novelty is low (it reuses existing VLM backbones), the training recipe, tokenization alignment, and unified action-language output space represent a paradigm-shifting simplification over prior modular or heavily structured robotics pipelines.
Results & evaluation
The experimental design is exceptionally rigorous, featuring ~6,000 real-world evaluation trials across in-distribution tasks, novel objects/backgrounds/environments, and three distinct emergent reasoning categories (symbol understanding, mathematical/multilingual reasoning, human recognition). The paper benchmarks against strong, diverse baselines (RT-1, VC-1, R3M, MOO) and demonstrates consistent 2x-6x generalization improvements. Ablations cleanly isolate the impact of model scale (5B vs 55B) and training regimes (scratch vs. fine-tune vs. co-fine-tune), conclusively proving the necessity of web-data co-training for preserving semantic capabilities. The inclusion of an open-source Language-Table simulation benchmark further strengthens the empirical claims and provides a reproducible testbed for the community.
Reproducibility
The paper provides clear algorithmic details: action discretization scheme, token mapping strategies (integer tokens for PaLI-X, least-frequent token overwriting for PaLM-E), co-fine-tuning data balancing, and output vocabulary constraints. However, full reproducibility is constrained by the reliance on proprietary, closed-weight VLMs (PaLI-X, PaLM-E) and large-scale cloud TPU infrastructure required for 1-5 Hz inference. While the training recipe is transparent, smaller labs cannot directly replicate the exact models without access to comparable compute and base model weights. The open-source simulation experiments partially mitigate this limitation.
Limitations
The authors correctly identify that VLA models do not acquire novel motor skills; physical capabilities remain strictly bounded by the distribution of the robot demonstration dataset. The approach also suffers from high computational overhead and low control frequencies (1-3 Hz for 55B models), making it unsuitable for high-frequency or safety-critical dynamic manipulation tasks. Furthermore, the dependency on proprietary VLMs and fine-tuning APIs creates a bottleneck for open research, and the paper does not address safety or failure modes when chain-of-thought reasoning produces semantically plausible but physically unsafe action sequences.
Broader impact
This work establishes Vision-Language-Action (VLA) models as a foundational paradigm for generalist robotics, demonstrating that Internet-scale pretraining can directly transfer to closed-loop physical control. It bridges the historical gap between semantic reasoning and low-level actuation, enabling zero-shot instruction following, multilingual commands, and rudimentary planning in real-world environments. The findings will likely accelerate industry and academic adoption of foundation models for robotics, though they also highlight growing concerns around compute centralization, hardware accessibility, and the need for robust safety frameworks when deploying open-ended reasoning agents in physical spaces. RT-2 introduces a unified tokenization and co-fine-tuning recipe that transforms large vision-language models into generalist robotic policies capable of zero-shot semantic reasoning and cross-domain generalization. By demonstrating that web-scale pretraining can be directly leveraged for low-level control without architectural modifications, the paper fundamentally shifts the robotics community's approach to data scaling, establishing VLA models as a dominant paradigm for embodied AI while clearly delineating current boundaries in skill acquisition, inference latency, and compute accessibility.
Curator note
Brohan et al.; Google; VLM directly outputs robot actions
Author of this paper? Send a correction.