Stereo-Planner: Learning Stereo-to-Trajectory Planning for Zero-Shot Sim-to-Real Quadrotor Navigation
Abstract
Existing navigation systems commonly rely on dedicated depth sensors or explicit depth reconstruction, while learning to generate trajectories directly from RGB images remains challenging due to the sim-to-real domain gap. In this work, we propose Stereo-Planner, an end-to-end vision-to-trajectory framework that enables zero-shot sim-to-real planning from stereo RGB observations. The key idea is to formulate trajectory generation as state- and scale-conditioned querying of latent stereo geometry. A Scene Representor incorporates a pretrained vision grounding foundation model to extract geometric cues shared across simulated and real-world environments, supporting generalization across visual domains. A lightweight Trajectory Decoder then translates these representations into trajectories through interleaved attention, separating trajectory-state interaction from scene-feature retrieval and incorporating the stereo baseline as a metric-scale cue. With these designs, Stereo-Planner is trained entirely on simulated data by minimizing differentiable trajectory costs and deployed directly in real-world environments without fine-tuning. Benchmark results demonstrate strong cross-scene generalization, with a 95% success rate in simulation environments at a commanded speed of 8 m/s, and competitive performance against depth-based planners using only stereo RGB observations for scene perception. Furthermore, deployment on a compact quadrotor equipped with an NVIDIA Orin NX enables autonomous flight at speeds of up to 7 m/s, with a mean onboard vision-to-action inference time of 24.27 ms. The software, simulation environments, and hardware designs will be open-sourced at https://github.com/Stereo-Planner/Stereo-Planner.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Stereo-Planner enables zero-shot sim-to-real quadrotor navigation using stereo RGB by leveraging a vision grounding foundation model for geometry-aware scene representation and a scale-conditioned trajectory decoder. The paper presents a significant technical contribution by demonstrating that pretrained visual features can effectively bridge the sim-to-real domain gap for end-to-end motion planning, achieving high-speed autonomous flight without depth sensors or fine-tuning.
Methodology
The paper proposes Stereo-Planner, an end-to-end framework that maps stereo RGB images directly to quadrotor trajectories. The core architectural contribution is the use of a pretrained vision grounding foundation model (Depth Anything V3 encoder) as a scene representor, which extracts geometry-aware features robust to sim-to-real appearance gaps. A lightweight trajectory decoder uses interleaved attention to query these features, conditioned on the MAV state and, crucially, the stereo baseline to resolve scale ambiguity. The training strategy employs differentiable trajectory costs (safety, smoothness, goal-reaching) and a Virtual Baseline Agmentation (VBA) technique to teach the network to interpret metric scale from varying stereo baselines without re-rendering environments. This approach effectively decouples geometric perception from appearance, allowing zero-shot transfer from simulation to reality.
Results & evaluation
The evaluation is rigorous, comparing against strong baselines like YOPO V2 and Ego-Planner V2 in both simulation and real-world deployment. The paper demonstrates a 95% success rate in unseen simulated environments (forest, industrial) and successful real-world flight at 7 m/s. Ablation studies clearly isolate the contributions of the foundation model initialization, the interleaved attention mechanism, and the VBA strategy. The comparison with depth-based methods shows that stereo RGB can achieve competitive performance without the hardware cost and latency of depth sensors.
Reproducibility
The authors commit to open-sourcing the software, simulation environments, and hardware designs. The paper provides detailed hyperparameters, loss function definitions, and architectural specifics. The use of standard Unity simulation and ROS interfaces enhances reproducibility. The specific version of the foundation model (DA3-Small) and its integration details are clearly described.
Limitations
The method relies on the availability of a suitable vision grounding foundation model (DA3), which may not be universally available or optimal for all tasks. The real-world evaluation, while impressive, is limited to specific environments (forest, industrial) and may not generalize to all cluttered scenarios (e.g., dynamic obstacles, extreme weather). The computational cost, while reduced, still requires an NVIDIA Orin NX, which is a significant hardware requirement for "compact" drones.
Broader impact
This work bridges the gap between high-level visual foundation models and low-level robotic control, a critical area for autonomous systems. By demonstrating that RGB-only stereo vision can replace depth sensors for agile navigation, it reduces hardware costs and increases the robustness of aerial robots. The methodology of using foundation model features for sim-to-real transfer in control tasks is likely to inspire similar approaches in other robotic domains. Stereo-Planner enables zero-shot sim-to-real quadrotor navigation using stereo RGB by leveraging a vision grounding foundation model for geometry-aware scene representation and a scale-conditioned trajectory decoder. The paper presents a significant technical contribution by demonstrating that pretrained visual features can effectively bridge the sim-to-real domain gap for end-to-end motion planning, achieving high-speed autonomous flight without depth sensors or fine-tuning.
Author of this paper? Send a correction.