JEPA4R: Embodied JEPA World Models with 4D Point Planning
Abstract
Joint-embedding predictive architecture (JEPA) world models predict future embeddings instead of pixels and plan in latent space. Two challenges remain: 2D images leave the metric geometry that manipulation acts on implicit and entangled with appearance, and what these models imagine stays hidden, as they are mostly judged in simulation by short-horizon latent planning. We present JEPA4R, a JEPA world model that observes single-depth-camera point clouds, predicts and plans in their latent space, and acts through a policy head on its frozen latents; we call this 4D point planning. A decoder on the frozen world model shows what it imagines as 4D point clouds, with a mean point error within 1.1 cm over 200 steps. Under the policy-learning protocols of OGBench and CortexBench, JEPA4R leads pixel JEPA models on OGBench Cube-Single, Cube-Double, Cube-Triple and Scene, and ranks first on MetaWorld and Adroit Relocate with 98.4% and 88.0%, above the runner-up, R3M, at 96.0% and 65.3%. Trained only in a digital twin, JEPA4R and its policy heads run zero-shot on the real arm with both a gripper and a dexterous hand.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
JEPA4R introduces a JEPA world model operating on 3D point clouds that outperforms 2D pixel-based models in manipulation tasks and enables visual verification of latent predictions through 4D point cloud decoding. The paper makes a strong case for the necessity of metric 3D geometry in embodied AI, providing rigorous evidence that 3D observations lead to better policy learning and sim-to-real transfer than 2D images, while offering a novel interpretability tool for latent space world models.
Methodology
The paper proposes JEPA4R, a world model that adapts the Joint-Embedding Predictive Architecture (JEPA) to operate on 3D point clouds rather than 2D images. The core methodological contribution is the "4D point planning" framework, which involves encoding single-depth-camera point clouds into a latent space using a PointNet-based encoder, predicting future latents using a transformer predictor conditioned on actions, and acting via a policy head on these frozen latents. A significant secondary contribution is the post-hoc decoder that reconstructs 4D point clouds from the imagined latents, allowing for visual verification of the world model's internal state. The approach is sound, leveraging established components (PointNet, Transformer, CEM planning) but applying them to a novel input modality (metric 3D geometry) to address the entanglement of appearance and geometry in 2D representations.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors compare JEPA4R against state-of-the-art 2D JEPA models (PLDM, LeWM, V-JEPA 2.1, etc.) on established benchmarks (OGBench, CortexBench/MetaWorld, Adroit). JEPA4R demonstrates superior performance, particularly on tasks requiring precise 3D manipulation (Cube tasks, Adroit Relocate). The inclusion of real-world zero-shot transfer from a digital twin to a physical arm (RM75-B) with both a gripper and a dexterous hand is a strong validation of the sim-to-real gap closure provided by 3D observations. Ablation studies on decoder backbones, encoders, and hyperparameters further support the robustness of the findings.
Reproducibility
The paper provides high reproducibility. Code and a project page are linked. Detailed implementation specifics are given, including network architectures (PointNet variant, ViT-tiny for baselines), training hyperparameters (AdamW, lr 5e-5, batch 128), and data collection protocols (paired 2D/3D data from simulation). The use of standard benchmarks and clear evaluation metrics (success rate, point error) allows for easy replication.
Limitations
The primary limitation is the reliance on a single depth camera and a specific digital twin setup, which may not generalize to all real-world environments without retuning or data collection. The paper acknowledges the scarcity of large-scale 3D robot data as a broader field limitation. Additionally, the decoder is post-hoc and not integrated into the training loop, which might limit the model's ability to optimize for reconstructive fidelity during learning, though the results show the JEPA objective inherently retains geometry.
Broader impact
This work has significant implications for the robotics and embodied AI community. It challenges the prevailing paradigm of using 2D pixels as the primary input for world models, suggesting that metric 3D geometry is superior for manipulation tasks. The ability to visualize the "imagination" of a latent world model via 4D point cloud decoding offers a new tool for debugging and safety verification in autonomous systems. This could lead to more robust and transparent robot control policies. JEPA4R introduces a JEPA world model operating on 3D point clouds that outperforms 2D pixel-based models in manipulation tasks and enables visual verification of latent predictions through 4D point cloud decoding. The paper makes a strong case for the necessity of metric 3D geometry in embodied AI, providing rigorous evidence that 3D observations lead to better policy learning and sim-to-real transfer than 2D images, while offering a novel interpretability tool for latent space world models.
Author of this paper? Send a correction.