OccJEPA: Occlusion as Masking for Amodal Video Segmentation
Abstract
Amodal video segmentation asks for the complete mask of an object in every frame, including the parts hidden behind other objects. The best current methods generate these masks with fine-tuned video diffusion models, which need many denoising steps and an extra depth network, and give a different answer for every sample. Self-supervised video world models of the JEPA family are trained on a closely related task: predicting the representation of masked space-time regions from the visible ones. We therefore treat occlusion as masking. The tokens that the object's visible mask does not cover are dropped, the pre-trained V-JEPA 2.1 predictor fills them in, and a light head decodes the observed and predicted features into amodal masks. The resulting model, OccJEPA (Occlusion-as-masking JEPA), is deterministic and 33.5x faster than the diffusion-based state of the art, and more accurate on SAIL-VOS, on TAO-Amodal, including a held-out test split (+4.00 AP50) and visible masks tracked by SAM2, and on Kubric. Controlled experiments trace the gain to the predictor's pre-trained weights rather than to its capacity, for world models from 87M to 1.85B parameters. Code is available at https://github.com/zgp110/occjepa.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces OccJEPA, a deterministic and highly efficient method for amodal video segmentation that leverages pre-trained JEPA world models by treating occlusion as masking, achieving state-of-the-art accuracy with a 33.5x speedup over diffusion-based approaches.
Methodology
The paper proposes OccJEPA, a method for amodal video segmentation that reframes occlusion as a masking problem solvable by pre-trained self-supervised video world models (specifically V-JEPA 2.1). Instead of using computationally expensive diffusion models with depth estimation, the method drops tokens corresponding to visible object masks and uses the JEPA predictor to fill in the latent representations of the occluded regions. A lightweight head then decodes these features into amodal masks. The core insight is that the predictive capability of JEPA models, trained to reconstruct masked spatiotemporal regions, naturally aligns with the task of inferring hidden object parts. The approach is deterministic, avoiding the stochasticity of diffusion sampling, and leverages the robustness of pre-trained weights rather than fine-tuning heavy generative models.
Results & evaluation
The experimental section demonstrates significant improvements over state-of-the-art diffusion-based methods. The model achieves a 33.5x speedup while maintaining or improving accuracy on SAIL-VOS, TAO-Amodal (including a held-out test split with +4.00 AP50), and Kubric. Ablation studies confirm that the performance gain stems from the pre-trained weights of the JEPA predictor rather than its architectural capacity, showing consistent results across model sizes from 87M to 1.85B parameters. The use of SAM2 for visible mask tracking ensures a fair comparison with other methods that rely on similar initialization.
Reproducibility
The authors provide a public GitHub repository, which enhances reproducibility. The reliance on standard pre-trained models (V-JEPA 2.1) and established datasets (SAIL-VOS, TAO, Kubric) further supports reproducibility. However, the specific details of the "light head" architecture and training hyperparameters would need to be verified in the code repository for full replication.
Limitations
The method relies on the quality of the visible masks provided by SAM2; errors in visible segmentation may propagate to the amodal prediction. The approach is tested primarily on standard benchmarks and may not generalize to highly dynamic or complex occlusion scenarios not covered by the training data of the pre-trained JEPA model. The deterministic nature, while a strength for consistency, may limit the exploration of multiple plausible amodal configurations in ambiguous cases.
Broader impact
This work bridges the gap between self-supervised representation learning and dense prediction tasks, suggesting that world models can be effectively repurposed for segmentation without generative overhead. The significant speedup makes amodal segmentation more practical for real-time applications. The finding that pre-trained weights are more critical than capacity for this task has implications for the design of future video understanding models. The paper introduces OccJEPA, a deterministic and highly efficient method for amodal video segmentation that leverages pre-trained JEPA world models by treating occlusion as masking, achieving state-of-the-art accuracy with a 33.5x speedup over diffusion-based approaches.
Author of this paper? Send a correction.