Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes
Abstract
City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large outdoor scenes. We propose Urbex, an agentic vision-language framework that instead casts grounding as active spatial search. Urbex represents a city-scale 3D scene as an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit a grounded bounding box. We optimize this tool-use policy with reinforcement learning on CityRefer, using smooth localization rewards and lightweight shaping rewards for efficient exploration. Experiments show that Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer further indicate improved cross-scene robustness.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Urbex, an agentic framework that reformulates city-scale 3D grounding as active spatial search using a VLM agent and reinforcement learning. It demonstrates superior accuracy and efficiency compared to candidate-ranking baselines, offering a scalable solution for large outdoor 3D scene understanding.
Methodology
The paper proposes "Urbex," a framework that reframes city-scale 3D grounding from a static candidate ranking problem to an active spatial search problem. The core novelty lies in the design of an interactive multi-view environment where a Vision-Language Model (VLM) agent uses specific tools (Landmark Query, Zoom-in, Oblique Rendering) to navigate the scene. The use of Reinforcement Learning (specifically GRPO) to optimize this tool-use policy is a strong methodological contribution, addressing the challenge of credit assignment in multi-step agentic tasks. The reward design, combining terminal IoU with distance-based shaping and tool-level rewards, is well-motivated for sparse reward environments. The approach effectively leverages the reasoning capabilities of modern VLMs while constraining the action space to manageable geometric operations.
Results & evaluation
The experiments are rigorous and comprehensive. The authors evaluate on CityRefer (in-domain) and STPLS3D-Refer (zero-shot cross-scene), covering both "With GT" (candidate ranking) and "Without GT" (candidate-free) settings. The results demonstrate significant improvements in Hit@1 and IoU over strong baselines like CityAnchor and GPT-5.4/Qwen3.6-plus prompted baselines. The efficiency analysis is particularly compelling, showing a 10x speedup over the strongest ranking baseline by avoiding exhaustive scoring. Ablation studies clearly isolate the contributions of individual tools and reward components, providing strong evidence for the design choices. The zero-shot generalization results are a key strength, suggesting the learned policy captures reusable spatial search strategies rather than just memorizing the training distribution.
Reproducibility
The paper provides detailed descriptions of the environment, tools, and RL objective. However, as an arXiv preprint, specific code availability is not confirmed in the text. The reliance on specific VLM backbones (Qwen, GPT) and the complexity of the RL training loop (GRPO) may pose barriers to reproduction for groups without significant compute resources. The hyperparameters for the RL training and the specific implementation of the view-to-world transforms are described but would benefit from open-source code for full verification.
Limitations
The method relies on the availability of named landmarks or distinct visual features for the "Landmark Query" tool, which may limit its applicability to scenes with poor semantic labeling or highly repetitive structures. The RL training process is computationally expensive compared to supervised fine-tuning. The "Oblique Rendering" tool requires pre-computed or on-the-fly rendering, which adds latency and storage overhead, although the paper addresses this with caching strategies. The performance on "Without GT" settings, while better than baselines, still shows room for improvement in precise box localization (high area ratio variance).
Broader impact
This work has significant implications for embodied AI and autonomous driving, where efficient and accurate localization of objects in large-scale environments is critical. By demonstrating that agentic search can outperform and outpace exhaustive ranking, it offers a scalable paradigm for handling large 3D scenes. The framework could be extended to other tasks requiring long-range spatial reasoning, such as robot navigation or aerial inspection. The integration of RL with VLM tool-use is a trend that this paper advances meaningfully in the 3D domain. The paper introduces Urbex, an agentic framework that reformulates city-scale 3D grounding as active spatial search using a VLM agent and reinforcement learning. It demonstrates superior accuracy and efficiency compared to candidate-ranking baselines, offering a scalable solution for large outdoor 3D scene understanding.
Author of this paper? Send a correction.