GATOR: Generative and Agentic 3D Object Reconstruction From Casual Images
Abstract
Reconstructing complete, scene-aligned 3D objects from casual images requires integrating sparse, uncertain observations and inferring surfaces hidden by occlusions. We present GATOR, a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images. Our local modality mixer couples patch-aligned RGB, target-mask, and pointmap features before cross-view reasoning, preserving scene context while distinguishing the target from its surroundings. Text-guided semantic conditioning complements these spatial cues with category names and object captions through stage-specific adapters for structure, geometry, and appearance generation. The generated asset initializes a multimodal agent, providing instance-specific geometry and pose for targeted structural and texture refinement through an observation-guided edit-render-review loop. Across synthetic objects, cluttered tabletops, and indoor scenes, GATOR achieves strong geometric and appearance fidelity while recovering scene-relative pose from sparse observations. Time-budget comparisons and scene-level simulation further demonstrate the reconstruction efficiency and simulation readiness. Project page: https://research.nvidia.com/labs/lpr/gator/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GATOR introduces a novel generative-agentic framework for 3D object reconstruction that couples a modality mixer for initial asset generation with an iterative agentic refinement loop, achieving strong geometric and appearance fidelity from casual images while recovering scene-relative pose, though the lack of detailed quantitative results in the provided text limits the full assessment of its empirical superiority over existing feed-forward and iterative baselines.
Methodology
The paper proposes GATOR, a framework combining a "local modality mixer" for feature fusion (RGB, mask, pointmap) with a "multimodal agent" for iterative refinement. The core novelty lies in the agentic loop (edit-render-review) that uses the initial generative output to guide targeted structural and texture corrections. This hybrid approach of one-shot generation followed by agentic refinement is a significant architectural shift from purely feed-forward reconstruction models. The use of stage-specific adapters for text-guided conditioning is a standard but effective technique. The methodology is sound, leveraging recent trends in agentic AI for 3D tasks, though the specific implementation details of the "agent's" decision-making process (e.g., how it selects which parts to refine) are critical to its success and likely complex.
Results & evaluation
The paper claims strong geometric and appearance fidelity on synthetic objects, cluttered tabletops, and indoor scenes. It includes time-budget comparisons and scene-level simulation readiness tests. However, the provided text is a placeholder with section headers but no actual quantitative results, tables, or figures. Without specific metrics (e.g., Chamfer Distance, FID, PSNR) or comparisons to state-of-the-art baselines (like Zero-1-to-3, Wonder3D, or recent agentic 3D papers), the experimental evaluation cannot be fully verified. The claim of "strong fidelity" is unsubstantiated in the provided text.
Reproducibility
The project page is provided, which is a positive sign for reproducibility. However, the lack of detailed hyperparameters, agent prompt engineering details, and training data specifics in the provided text makes independent reproduction difficult. The "agentic" component often relies on proprietary LLMs or specific prompt strategies that may not be fully open-sourced.
Limitations
The primary limitation is the reliance on an agentic loop, which can be computationally expensive and slow compared to feed-forward methods, despite the "time-budget" claims. The performance on real-world, uncontrolled casual images (as opposed to synthetic or curated datasets) is not clearly demonstrated in the provided text. The "scene-aligned" pose recovery from sparse observations is a difficult problem, and the robustness of this component to severe occlusions or ambiguous viewpoints is a potential weakness.
Broader impact
This work has high potential impact in robotics, AR/VR, and digital twin creation, where accurate 3D asset recovery from casual photos is crucial. The agentic approach could be extended to other 3D tasks like scene editing or animation. It represents a step towards more interactive and intelligent 3D reconstruction pipelines. GATOR introduces a novel generative-agentic framework for 3D object reconstruction that couples a modality mixer for initial asset generation with an iterative agentic refinement loop, achieving strong geometric and appearance fidelity from casual images while recovering scene-relative pose, though the lack of detailed quantitative results in the provided text limits the full assessment of its empirical superiority over existing feed-forward and iterative baselines.
Author of this paper? Send a correction.