ActionCodec2: A Streaming Action Tokenizer
Abstract
Most action tokenizers for autoregressive vision language action (VLA) models compress motion into chunks that must be decoded in full before their actions become executable. We introduce ActionCodec2, a non-neural streaming tokenizer that releases actions incrementally while keeping trajectory reconstruction error within a prescribed tolerance. Its encoder quantizes each step relative to the decoder's reconstructed state, so the selected motion primitive corrects residual error rather than allowing errors from independently quantized increments to accumulate. Finding the smallest such primitive set is a geometric covering problem; we derive the exact minimum for translation and constant-factor-tight bounds for rotation. Set-BPE then losslessly merges co-occurring primitive symbols across coordinates and neighboring steps, without a fixed coordinate ordering. A universal version fitted on 20k hours of data from 24 embodiments averages 8.8 tokens per second for 15 Hz end-effector motion on this corpus. With the same VLA backbone, ActionCodec2 outperforms the compared tokenizers on three simulation benchmarks and in zero-shot real-robot evaluation. Its streaming representation allows policies to incorporate state feedback within an action chunk, substantially improving performance over open-loop execution under randomized dynamics, external disturbances, and moving objects. The same tokens also can serve as a direct control interface for VLM agents, which compose successful action sequences into reusable macros. These agent-built macros reduce control decisions and thinking-token usage across attempts and transfer to a held-out task. More results are available at https://zibindong.github.io/actioncodec2-page/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ActionCodec2 introduces a streaming, non-neural action tokenizer that guarantees bounded tracking error via geometric covering and lossless Set-BPE compression, enabling state-feedback within action chunks and serving as a direct control interface for VLM agents. The paper presents a rigorous theoretical foundation for action tokenization, offering exact solutions for translation and tight bounds for rotation, which distinguishes it from heuristic or learned approaches. By decoupling the tokenizer from neural optimization and providing a universal, plug-and-play solution, it addresses key scalability and robustness challenges in autoregressive VLA models. The experimental results, particularly the improvement in closed-loop execution and the agent-based macro learning, demonstrate significant practical utility and suggest a new paradigm for integrating discrete action tokens with high-level reasoning.
Methodology
The paper proposes ActionCodec2, a non-neural, streaming action tokenizer for autoregressive Vision-Language-Action (VLA) models. The core innovation is a geometric covering approach where the encoder quantizes each step relative to the decoder's reconstructed state, ensuring that tracking error remains within a prescribed tolerance rather than accumulating. The authors derive exact minimum primitive sets for translation and tight bounds for rotation. Additionally, they introduce Set-BPE, a lossless compression algorithm that merges co-occurring primitive symbols across coordinates and time steps without fixed coordinate ordering. The method is entirely deterministic and requires no gradient-based optimization for fitting the tokenizer itself, which is a significant practical advantage.
Results & evaluation
The experiments are extensive, covering three simulation benchmarks (LIBERO, LIBERO-Plus, SimplerEnv) and zero-shot real-robot evaluation on a Franka arm. The paper demonstrates that ActionCodec2 outperforms strong baselines like FAST, X-Tokenizer, and OAT. A particularly strong result is the demonstration of "closed-loop" execution within a chunk, where state feedback is inserted between action tokens, significantly improving robustness to disturbances and moving objects compared to open-loop execution. The agent interface experiments, where VLMs use the tokens to build reusable macros, are a novel and compelling application.
Reproducibility
The paper provides high reproducibility. The tokenizer fitting process is described in detail (Algorithms 1 and 2), and the authors release pretrained universal tokenizers on Hugging Face. The code is available on GitHub. The non-neural nature of the tokenizer fitting makes it highly reproducible compared to learned VQ-VAE approaches.
Limitations
The primary limitation is the computational cost of the Set-BPE fitting process on large corpora, although the authors note it runs on a single CPU. The method is optimized for end-effector speeds up to ~1.2 m/s; highly dynamic motions may see reduced compression ratios compared to chunk-based tokenizers. The real-robot evaluation, while zero-shot, is limited to a single robot embodiment (Franka) and a specific set of tasks.
Broader impact
This work has high potential impact on the VLA community by providing a robust, interpretable, and efficient action representation. The ability to insert state feedback within an action chunk addresses a critical limitation of current AR VLAs, potentially enabling more reactive and robust robot policies. The use of tokens as a direct interface for VLM agents opens new avenues for high-level planning and skill reuse. ActionCodec2 introduces a streaming, non-neural action tokenizer that guarantees bounded tracking error via geometric covering and lossless Set-BPE compression, enabling state-feedback within action chunks and serving as a direct control interface for VLM agents. The paper presents a rigorous theoretical foundation for action tokenization, offering exact solutions for translation and tight bounds for rotation, which distinguishes it from heuristic or learned approaches. By decoupling the tokenizer from neural optimization and providing a universal, plug-and-play solution, it addresses key scalability and robustness challenges in autoregressive VLA models. The experimental results, particularly the improvement in closed-loop execution and the agent-based macro learning, demonstrate significant practical utility and suggest a new paradigm for integrating discrete action tokens with high-level reasoning.
Author of this paper? Send a correction.