TartanMatch: Towards Universal Dense Correspondence Across Modalities
Abstract
Robotic platforms combine multi-modal sensors to operate across diverse conditions, but most dense correspondence methods are limited to RGB-RGB or a small set of modality pairs. Training a model for each modality pair scales poorly with the number of sensors and forgoes potential positive transfer across modality pairs. We introduce TartanMatch, a unified model for dense correspondence across all 25 ordered pairs among RGB, event, thermal, depth, and projected LiDAR. TartanMatch adapts an RGB-pretrained correspondence model to all modality pairs within one shared network using lightweight modality adaptation and a mixture of real and synthetic data. On zero-shot real-world benchmarks, TartanMatch reduces Endpoint Error (EPE) over the strongest baselines by 61.9% for cross-modal matching and 49.6% for same-modal matching. We further show that joint training across all modality pairs improves average matching accuracy over pair-specific training, especially on challenging cross-modal pairs. These results demonstrate that a single model achieves broad multi-modal coverage and state-of-the-art matching accuracy.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
TartanMatch introduces a unified dense correspondence model that significantly outperforms pair-specific baselines across 25 modality pairs by leveraging RGB pre-training and joint multi-modal training, demonstrating that a single network can effectively handle diverse sensor combinations with superior accuracy and scalability.
Methodology
The paper proposes TartanMatch, a unified framework for dense correspondence across five distinct sensing modalities: RGB, event, thermal, depth, and projected LiDAR. The core methodological contribution is the adaptation of an RGB-pretrained correspondence model (specifically UFM) to handle all 25 ordered modality pairs within a single shared network. This is achieved through lightweight modality adaptation modules and a training strategy that leverages a mixture of real and synthetic data. The approach addresses the scalability issue of training separate models for each modality pair, arguing that joint training facilitates positive transfer, particularly for challenging cross-modal pairs. The use of a pre-trained backbone for RGB and adapting it to other modalities is a sound architectural choice that leverages the rich features learned from large-scale RGB datasets.
Results & evaluation
The experimental evaluation is extensive, covering zero-shot real-world benchmarks. The paper reports significant improvements in Endpoint Error (EPE), claiming a 61.9% reduction for cross-modal matching and 49.6% for same-modal matching compared to the strongest baselines. The radar chart visualization effectively demonstrates the model's superiority across various modality pairs. The claim that joint training improves average matching accuracy over pair-specific training is a key empirical finding that supports the unified model hypothesis. The inclusion of both real and synthetic data in the training mix is well-justified by the results, showing robustness to domain shifts.
Reproducibility
The paper provides a project page (https://tartanmatch.github.io/) which likely contains code and additional resources. The use of standard benchmarks and clear metrics (EPE) enhances reproducibility. However, the specific details of the "lightweight modality adaptation" modules and the exact composition of the synthetic data mixture would need to be verified in the supplementary material or code repository for full reproducibility. The reliance on a specific pre-trained model (UFM) is a potential dependency, but UFM is a known baseline.
Limitations
The primary limitation is the scope of modalities covered (5 modalities, 25 pairs). While this is a significant expansion over previous work, it does not cover all possible robotic sensing modalities (e.g., radar, acoustic). The performance on zero-shot real-world data is strong, but the gap between synthetic/real training data and real-world inference might still exist for unseen environments. The computational cost of training a single model for 25 pairs versus 25 separate models is not explicitly detailed in the abstract, though the "lightweight" adaptation suggests efficiency.
Broader impact
This work has high potential impact in the robotics and autonomous systems fields. A unified correspondence model simplifies the perception pipeline for multi-modal robots, reducing the need for maintaining and updating multiple specialized models. The demonstrated positive transfer across modalities could inspire similar unified approaches in other multi-sensor fusion tasks. The open-source nature of the project (implied by the project page) will likely facilitate adoption by the research community. TartanMatch introduces a unified dense correspondence model that significantly outperforms pair-specific baselines across 25 modality pairs by leveraging RGB pre-training and joint multi-modal training, demonstrating that a single network can effectively handle diverse sensor combinations with superior accuracy and scalability.
Author of this paper? Send a correction.