TAPESTRY: A Single Backbone for Video Generation and Robot Control
Abstract
Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
Methodology
The paper proposes TAPESTRY, a method to repurpose pretrained video diffusion transformers (specifically CogVideoX-2B) for robot control by treating robot actions as single-patch tokens in the video latent space. The core innovation is the "action-as-VAE-patch" tokenization, which uses a lightweight linear projection and LayerNorm to map action vectors into the same distribution as video VAE latents, allowing them to pass through the pretrained patchifying convolution without architectural modification. The training procedure involves three joint diffusion objectives: video-only, joint video+action, and action-only. A significant technical contribution is the analysis of high-frequency noise in multi-step denoising for action chunks, demonstrating that single-step DDIM inference from pure noise is superior for this specific repurposed backbone due to frequency-space mismatches between video and action signals.
Results & evaluation
The evaluation is rigorous, covering simulation (LIBERO benchmark with a learning-from-video protocol) and real-world bimanual manipulation. The paper demonstrates strong generalization to novel objects and motion primitives. The inference efficiency claims are well-supported, showing a 44x speedup over optimized DreamZero on consumer hardware (RTX 5090). The ablations effectively isolate the contribution of the video pretraining and the specific training objectives.
Reproducibility
The paper provides detailed implementation specifics, including the exact parameter count of the action head (1280 parameters), the specific backbone used (CogVideoX-2B), and the training data mixture. The derivation of the single-step inference collapse is provided in the appendix. However, reliance on specific proprietary or large-scale datasets (like DROID, AgiBot-World) may limit immediate reproducibility for smaller labs, though the architectural changes are minimal and easy to implement.
Limitations
The method is currently limited to a 20-DoF bimanual action space, and the single-step inference relies on the zero-terminal-SNR v-prediction schedule, which may not transfer to other backbones like rectified-flow models (Wan). The deterministic nature of single-step inference may be a limitation for tasks requiring multimodal action distributions.
Broader impact
This work significantly lowers the barrier to entry for using large video models in robotics by eliminating the need for expensive video generation at test time. The insight that action tokens can be treated as video patches with minimal projection is a generalizable architectural pattern that could be applied to other multimodal control tasks. The finding regarding high-frequency noise in repurposed diffusion models is a valuable diagnostic for the broader community working on world action models. TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
Author of this paper? Send a correction.