ProMo3D: Probing Motion Cues in Frozen 3D Foundation Models
Abstract
Feed-forward 3D foundation models have recently reshaped multi-view reconstruction, yet dynamic 4D understanding still often relies on large-scale motion supervision or hand-crafted, architecture-specific heuristics. We ask whether frozen 3D foundation model tokens already encode motion-relevant information that can be exposed by lightweight readout functions, rather than re-learning motion-specific features with high-capacity decoders. Through probing on three representative backbones, we find that static--dynamic separation is decodable from frozen token representations. Our rank-restricted and spectral analyses on $π^3$X reveal that the recovered motion cues are concentrated in a low-dimensional subspace. Building on this, we propose Probing-based Motion Distillation (PMD), which converts pairwise motion-state compatibility scores against a fixed anchor token into dense motion masks. Trained on only 160 YouTube-VOS videos using a single 16 GB GPU, PMD transfers zero-shot to DAVIS, SegTrackv2, and FBMS-59 without target-benchmark fine-tuning. With multi-view pretrained geometric backbones, PMD outperforms training-free heuristics and approaches substantially heavier supervised and self-supervised methods, suggesting that motion decoding from frozen 3D foundation models can be lightweight and data-efficient.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ProMo3D demonstrates that frozen 3D foundation models encode decodable motion cues in a low-dimensional subspace, enabling a lightweight, data-efficient motion segmentation method (PMD) that achieves competitive performance with heavy supervised models using only 160 training videos and a single GPU, thereby establishing a new paradigm for efficient 4D understanding via representation probing.
Methodology
The paper proposes Probing-based Motion Distillation (PMD), a lightweight framework that extracts motion cues from frozen 3D foundation models (specifically $\pi^3$X, DA3, and DUSt3R) using low-rank quadratic probes. The core methodological contribution is the demonstration that static-dynamic separation is decodable from frozen token representations without retraining the backbone. The authors introduce a pairwise classification formulation for motion-state compatibility, utilizing a low-rank bilinear probe to capture channel-wise interactions. This approach is theoretically grounded in representation probing literature and practically efficient, requiring only 160 training videos and a single consumer-grade GPU. The use of a fixed anchor token for unary score generation is a clever, albeit slightly heuristic, design choice that simplifies the decoding process significantly compared to full pairwise decoding.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors conduct layer-wise probing to identify where motion cues emerge in the backbone, providing valuable insights into the internal representations of 3D foundation models. Zero-shot generalization is tested across multiple benchmarks (DAVIS, SegTrackv2, FBMS-59) without fine-tuning, demonstrating strong data efficiency. The comparison against both training-free heuristics (Easi3R, VGGT4D) and heavy supervised methods (GeoMotion, Flow-SAM) is well-chosen, highlighting the performance-efficiency trade-off. The inclusion of ablations on probe rank, training data size, and backbone choice strengthens the empirical claims. The results show that PMD approaches the performance of much heavier supervised methods while using orders of magnitude less data and compute.
Reproducibility
The paper provides high reproducibility. All experiments are conducted on a single RTX 5070 Ti, making the setup accessible. Detailed hyperparameters, training protocols, and inference algorithms are provided in the appendix. The use of standard datasets and publicly available backbones further enhances reproducibility. The code and model weights are implied to be available or easily reconstructible given the detailed description of the probe architecture and training procedure.
Limitations
The primary limitation is the binary nature of the motion detection; PMD does not distinguish between different types of motion (e.g., articulation vs. displacement) or provide fine-grained kinematic information. There is also an issue of semantic leakage, where static instances of a moving category may receive elevated dynamic scores. The reliance on a fixed anchor token, while effective, is a potential vulnerability if the anchor region is dynamic, although the authors provide stress tests showing robustness to this. The evaluation is limited to specific 3D foundation models, and the generalizability to other architectures or modalities is not fully explored.
Broader impact
This work has significant implications for the field of 4D understanding and dynamic scene reconstruction. By demonstrating that motion cues are already encoded in frozen 3D foundation models, it suggests a new paradigm for lightweight, data-efficient motion decoding. This could reduce the computational and data requirements for dynamic perception tasks, making them more accessible for resource-constrained applications such as robotics and autonomous driving. The findings also contribute to the broader understanding of what information is captured by large-scale geometric pretraining, potentially guiding future pretraining objectives. ProMo3D demonstrates that frozen 3D foundation models encode decodable motion cues in a low-dimensional subspace, enabling a lightweight, data-efficient motion segmentation method (PMD) that achieves competitive performance with heavy supervised models using only 160 training videos and a single GPU, thereby establishing a new paradigm for efficient 4D understanding via representation probing.
Author of this paper? Send a correction.