Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles
Abstract
We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time. Our video inference pipeline uses pose graph optimization to refine camera trajectories and assembles object tracks across input windows. We also develop a data annotation pipeline and release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on Waymo and nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy among the compared methods. Our task-combination study shows benefits for both detection and geometry: co-training detection with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training, while co-training geometry with detection reduces camera-to-rig translation error by $13.4\%$ relative to geometry-only training. We further use the predicted geometry and trajectories as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Foundation4D introduces a unified feed-forward model for 4D scene understanding in autonomous driving, achieving state-of-the-art results in geometry, detection, and reconstruction. The paper makes a strong technical contribution through its factorized spatiotemporal attention mechanism and joint training strategy, supported by the release of a new benchmark and rigorous empirical validation of multi-task learning benefits.
Methodology
The paper proposes Foundation4D, a unified feed-forward model that jointly estimates geometry (depth), 3D detection, semantic occupancy, and Gaussian splatting reconstruction from multi-camera driving video. The core architectural contribution is a shared backbone utilizing factorized spatiotemporal attention to efficiently fuse observations across multiple camera views and temporal steps. This design allows the model to leverage temporal consistency for static scenes while handling dynamic objects. The method also introduces a video inference pipeline that employs pose graph optimization to refine camera trajectories and assembles object tracks across input windows, addressing the challenge of maintaining global consistency in long sequences. The joint training strategy is a key methodological strength, demonstrating that co-training detection with geometry and occupancy yields mutual benefits, improving both detection accuracy and geometric estimation precision.
Results & evaluation
The authors introduce PAI-Foundation4D, a new benchmark with dense depth and semantic occupancy labels for 1,253 driving clips, which is a significant contribution to the field. Experimental results show that Foundation4D achieves state-of-the-art performance on Waymo and nuScenes for depth estimation (lowest AbsRel), nuScenes for 3D detection (highest mAP), and OpenOccupancy for occupancy (highest mIoU). The ablation studies on task combination are particularly compelling, showing that co-training improves top-class mAP by 1.2 points and reduces camera-to-rig translation error by 13.4% compared to single-task baselines. The application to lidarless NuRec demonstrates the practical utility of the predicted geometry and trajectories for scene reconstruction in simulation.
Reproducibility
The paper releases the PAI-Foundation4D benchmark, which enhances reproducibility for the evaluation metrics. However, the full model architecture and training code are not explicitly stated to be released in the provided text, though the detailed description of the factorized attention and pose graph optimization provides a clear roadmap for implementation. The reliance on specific proprietary datasets (Waymo) may limit immediate reproducibility for external groups without access to those resources.
Limitations
The model is evaluated primarily on driving datasets, and its generalizability to other domains (e.g., robotics, indoor navigation) is not explored. The computational cost of the factorized spatiotemporal attention and pose graph optimization during inference may be high, potentially limiting real-time deployment on edge devices. The paper does not extensively discuss failure cases or robustness to extreme weather conditions or occlusions beyond standard benchmark evaluations.
Broader impact
This work has significant implications for the development of end-to-end autonomous driving systems by unifying perception and reconstruction tasks. The release of a new benchmark with dense labels will likely accelerate research in 3D scene understanding for autonomous vehicles. The demonstrated benefits of multi-task learning for geometry and detection provide valuable insights for practitioners designing unified perception stacks. The application to simulation via Gaussian reconstruction supports the growing trend of using neural rendering for training and testing autonomous driving algorithms. Foundation4D introduces a unified feed-forward model for 4D scene understanding in autonomous driving, achieving state-of-the-art results in geometry, detection, and reconstruction. The paper makes a strong technical contribution through its factorized spatiotemporal attention mechanism and joint training strategy, supported by the release of a new benchmark and rigorous empirical validation of multi-task learning benefits.
Author of this paper? Send a correction.