Generative Representation Learning for Scalable Latent Tokenization
Abstract
We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional collapse and feature suppression in cases where commonly used methods like DINO and LeJEPA fail. Moreover, we show our representation learning technique is scalable; expending more compute leads to predictably better latent tokenization. For the first time, we perform a careful analysis of the combined flop efficiency of tokenizer and generative model and demonstrate that our approach is a compute-optimal improvement over direct diffusion for pixel-space generation, obtaining an FID-50K of 2.18 with JiT-L. Applied to ImageNet and the text-to-image dataset GPIC, we show tokenizer encoder scaling laws for diffusability, where extrapolating our approach achieves representations in 7x fewer FLOPs than DINOv2 on ImageNet and 8x fewer FLOPs than WebSSL on GPIC.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces RePR, a generative representation learning method that factorizes Pointwise Mutual Information to create scalable, feature-suppression-robust tokenizers, demonstrating that this approach yields compute-optimal improvements over direct pixel diffusion for image generation.
Methodology
The paper proposes RePR (Relative Prediction Representations), a self-supervised representation learning method that explicitly factorizes the Pointwise Mutual Information (PMI) of the data using a generative model to estimate PMI. The core innovation is replacing the discriminative InfoNCE loss (which relies on negative sampling and is prone to feature suppression) with a generative objective that estimates PMI via the ELBO of a diffusion model. The method introduces "Timestep Weighting" to bias the PMI estimation toward semantically relevant noise scales and "Bootstrapped PMI Estimation" to make the likelihood estimation tractable for high-dimensional inputs by decomposing the PMI into incremental token predictions. The architecture uses a Masked Autoencoder (MAE) variant with a diffusion head for the prediction model and a separate representation encoder. The theoretical grounding is strong, linking the method directly to the information-theoretic properties of contrastive learning while avoiding its practical pitfalls.
Results & evaluation
The experiments are rigorous and address a critical gap in the field: the compute efficiency of tokenizers. The authors demonstrate that RePR encoders exhibit predictable scaling laws for "diffusability" (how well a generative model can learn from the latent space). They show that RePR achieves DINOv2-level performance with 7x fewer FLOPs on ImageNet and 8x fewer than WebSSL on GPIC. A key result is the demonstration that using RePR with Latent Forcing yields a better FID-50K (2.18) than direct pixel diffusion (JiT-L) at equal total training FLOPs, proving that representation learning is a compute-optimal component of the generative pipeline. The ablation studies on masking strategies, loss weights, and iterative feature prediction are thorough. The feature suppression experiment using watermarked images effectively highlights the robustness of the generative PMI objective compared to contrastive baselines like DINOv3 and LeJEPA.
Reproducibility
The paper provides a GitHub link and detailed implementation notes in the appendix, including architectural specifics (RoPE, register tokens, layer splits) and training hyperparameters. The use of standard components (ViT, MAE, Diffusion) aids reproducibility. However, the specific "Latent Forcing" method is cited as a separate work (Baade et al.), which may introduce dependency on external codebases not fully detailed here. The FLOP calculations are explicitly defined, which is crucial for the claims made.
Limitations
The method relies on a diffusion model for PMI estimation, which adds computational overhead during the representation learning phase, although the paper argues this is offset by downstream efficiency. The scaling laws are extrapolated, and the "7x/8x" claims depend on the accuracy of these power-law fits. The method is primarily evaluated on image generation; applicability to other modalities (audio, video) is mentioned but not extensively tested. The reliance on a specific masking strategy and timestep weighting scheme requires careful tuning.
Broader impact
This paper has high potential impact by providing a theoretically grounded, scalable alternative to contrastive representation learning for generative tasks. It challenges the prevailing assumption that discriminative objectives are sufficient for tokenizers and offers a path to more compute-efficient generative models. The demonstration of compute-optimal generation via learned representations could influence the design of future foundation models, encouraging a shift towards generative self-supervision for representation learning. The paper introduces RePR, a generative representation learning method that factorizes Pointwise Mutual Information to create scalable, feature-suppression-robust tokenizers, demonstrating that this approach yields compute-optimal improvements over direct pixel diffusion for image generation.
Author of this paper? Send a correction.