Scene-Adaptive VLA: Efficient Autonomous Driving via Dynamic Layer Routing
Abstract
Vision-Language-Action (VLA) models offer a promising paradigm for autonomous driving. However, the massive parameter scale of VLAs strains onboard computational resources during real-time inference. This directly increases energy consumption and limits vehicle battery range, making it critical to minimize computational overhead without compromising driving capability and explainability. Current research on efficient VLA-based driving focuses on sequential adaptation (\textit{e.g.}, Chain-of-Thought and token adjustment), leaving structural adaptation underexplored despite parameter redundancy across dynamic driving scenes. To address this gap, we propose Scene-Adaptive VLA, an efficient framework leveraging dynamic layer routing to adjust active parameters based on real-time scenes. Inspired by the temporal continuity of scenes and model predictive uncertainty, our approach characterizes model-perceived scene complexity via frame-level temporal scene variations and decision deviations. To uncover latent scene-to-parameter correspondences, we encode this characterization into scene-aware tokens, alongside two specialized queries. A budget-conditioned layer routing mechanism evaluates these refined queries to allocate a budget that determines the active parameter ratio and subsequently selects appropriate layer combinations. The entire framework is optimized end-to-end. Extensive evaluations on the Bench2Drive benchmark show that when adapted to a state-of-the-art VLA model, our approach significantly reduces inference-time computational overhead by 53.5\% with negligible degradation in driving and language capabilities. Code is available at https:github.com/yang-luvu/Scene-Adaptive-VLA.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a scene-adaptive layer routing mechanism for VLA models that reduces inference overhead by 53.5% while maintaining driving performance, offering a practical solution for deploying large multimodal models in resource-constrained autonomous driving systems.
Methodology
The paper proposes "Scene-Adaptive VLA," a framework for efficient inference in Vision-Language-Action models for autonomous driving. The core innovation is a structural adaptation mechanism that dynamically routes layers based on perceived scene complexity. The method characterizes complexity using temporal visual variations and decision deviations (uncertainty), encoding these into scene-aware tokens. A budget-conditioned routing mechanism uses these tokens to determine the active parameter ratio and specific layer combinations via a Gumbel-Softmax top-k sampling strategy. The training is two-stage: supervised fine-tuning for representation learning and GRPO (Group Relative Policy Optimization) for budget allocation. The methodology is sound, leveraging existing efficient inference concepts (early exit, MoE-like routing) but applying them specifically to the VLA driving domain with a novel complexity metric.
Results & evaluation
The experiments are conducted on the Bench2Drive benchmark, a standard for closed-loop autonomous driving evaluation. The paper reports a 53.5% reduction in FLOPs with negligible degradation in driving metrics (Driving Score, Success Rate) and language capabilities. Ablation studies confirm the contribution of each component (scene deviation, decision deviation, routing mechanism). Generalization is tested on a different backbone (ORION), showing similar efficiency gains. Safety-critical analysis indicates that collision rates do not increase, which is crucial for this domain. The results are strong and directly address the efficiency-performance trade-off.
Reproducibility
The paper provides a GitHub link to the code. Detailed implementation settings, hyperparameters, and training procedures are described in the appendix. The use of standard benchmarks (Bench2Drive, CARLA) and public models (SimLingo, ORION) enhances reproducibility. The specific configuration of the routing mechanism and loss functions is clearly defined.
Limitations
The approach is tested primarily on dense VLA models; applicability to sparse or MoE-native architectures is not fully explored. The computational overhead of the routing mechanism itself, while small, adds latency that might be significant in extremely low-latency requirements. The generalization to real-world hardware constraints beyond simulation is not tested. The reliance on temporal continuity for decision deviation assumes a certain frame rate and stability, which might not hold in all driving conditions.
Broader impact
This work contributes to the growing field of efficient AI for edge devices, particularly in safety-critical applications like autonomous driving. By demonstrating that significant computational savings can be achieved without compromising safety or performance, it encourages the adoption of VLA models in resource-constrained environments. The scene-adaptive routing concept could be extended to other multimodal tasks where input complexity varies significantly. The paper introduces a scene-adaptive layer routing mechanism for VLA models that reduces inference overhead by 53.5% while maintaining driving performance, offering a practical solution for deploying large multimodal models in resource-constrained autonomous driving systems.
Author of this paper? Send a correction.