Camera3R: Feed-Forward 3D Reconstruction from Heterogeneous Camera Models
Abstract
Feed-forward 3D reconstruction (3R) from unconstrained multi-view images has recently emerged as a prominent research direction in 3D computer vision. Various foundational reconstruction models have been proposed with different architectures and reconstruction paradigms. Despite being trained on large-scale perspective datasets with substantial computational resources, these models generally struggle to generalize to non-perspective images. This limitation hinders their deployment in applications involving arbitrary, unknown, and potentially heterogeneous camera models. To build on the capabilities of existing reconstruction models with minimal retraining, we propose Camera3R, a low-rank adaptation framework that efficiently extends perspective-pretrained 3R networks to any other camera models. Extensive experiments demonstrate that, with only 0.5% additional parameters and less than a day of training, Camera3R adapts a foundational 3D reconstruction model to non-perspective cameras while fully preserving its original performance on perspective inputs. Code is available at https://github.com/android-xr/camera3r.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Camera3R, a parameter-efficient LoRA-based framework that extends perspective-pretrained feed-forward 3D reconstruction models to heterogeneous non-perspective cameras (fisheye, panoramic) while preserving original performance. By employing a Mixture-of-Experts strategy with camera-specific adapters and a post-hoc rank truncation scheme, the method achieves state-of-the-art results on mixed-camera reconstruction tasks with minimal computational overhead, offering a practical solution for deploying 3D reconstruction in real-world systems with diverse sensor configurations.
Methodology
The paper proposes Camera3R, a parameter-efficient framework using Low-Rank Adaptation (LoRA) to extend perspective-pretrained feed-forward 3D reconstruction models (like VGGT, MapAnything, and $^3$) to non-perspective cameras (fisheye, panoramic). The core innovation is a Mixture-of-Experts (MoE) approach where specific LoRA experts are dedicated to specific camera models, while the perspective backbone remains frozen. This ensures backward compatibility. A key technical contribution is the "post-hoc adaptive rank truncation" scheme, which uses SVD to compress the LoRA matrices based on a global threshold, achieving a 22x compression with minimal performance loss. The method also includes a lightweight camera router to handle unknown camera types at inference time. The methodology is sound, leveraging established techniques (LoRA, MoE) in a novel and effective combination for this specific domain problem.
Results & evaluation
The experiments are extensive and rigorous. The authors evaluate on three different foundational backbones, demonstrating the generality of the approach. They test on multiple datasets spanning indoor (ScanNet++, Matterport3D) and outdoor (KITTI360, PanoCity) environments, covering perspective, fisheye, panoramic, and mixed-camera scenarios. The results show significant improvements over baselines like Wid3R, Fisheye3R, and PanoVGGT. The ablation studies clearly demonstrate the importance of separating experts per camera type and the effectiveness of the rank truncation. The claim of "less than a day of training" (20 hours on 8 GPUs) is verified in the appendix. The preservation of perspective performance is a strong selling point, clearly demonstrated in the results.
Reproducibility
High. The code is available on GitHub. The paper provides detailed hyperparameters, dataset splits, augmentation strategies, and hardware requirements in the appendix. The training procedure is well-defined. The use of standard open-source backbones (VGGT, etc.) and datasets further enhances reproducibility.
Limitations
The method assumes a predefined set of camera categories (perspective, fisheye, panoramic). Adding a new camera type requires training a new expert and updating the router, though this is claimed to be incremental. The training relies on synthetic cross-camera augmentation, which may not fully capture real-world sensor noise and lens imperfections beyond the modeled distortions. The method is specific to feed-forward reconstruction models; it may not directly apply to optimization-based methods like SfM/MVS pipelines without significant adaptation.
Broader impact
This work has high practical impact for AR/VR and autonomous driving applications where heterogeneous camera systems are common. By enabling existing large-scale models to handle non-perspective inputs with minimal retraining, it lowers the barrier to deploying advanced 3D reconstruction in real-world, mixed-camera setups. The efficiency (0.5% parameters, <1 day training) makes it attractive for industry adoption. It bridges the gap between academic perspective-only models and real-world heterogeneous sensor arrays. The paper introduces Camera3R, a parameter-efficient LoRA-based framework that extends perspective-pretrained feed-forward 3D reconstruction models to heterogeneous non-perspective cameras (fisheye, panoramic) while preserving original performance. By employing a Mixture-of-Experts strategy with camera-specific adapters and a post-hoc rank truncation scheme, the method achieves state-of-the-art results on mixed-camera reconstruction tasks with minimal computational overhead, offering a practical solution for deploying 3D reconstruction in real-world systems with diverse sensor configurations.
Author of this paper? Send a correction.