Masked Autoencoders Are Scalable Vision Learners (MAE)
Abstract
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper demonstrates that a simple, asymmetric masked autoencoder with high patch masking ratios is a highly scalable and effective paradigm for self-supervised vision pre-training. By decoupling the heavy encoder from the reconstruction decoder and proving that pixel-level targets suffice for learning transferable representations, the work fundamentally shifts the field away from complex contrastive and tokenization-based pipelines, establishing a new standard for efficient, scalable, and high-performing vision foundation models.
Methodology
The paper introduces a remarkably elegant solution to a long-standing bottleneck in vision self-supervised learning: the computational and representational inefficiency of applying BERT-style masked modeling to images. The core methodological innovation is the asymmetric encoder-decoder architecture. By restricting the heavy encoder to only visible patches (completely omitting mask tokens) and delegating reconstruction to a lightweight decoder, the authors reduce pre-training compute by ~3x while preserving representational capacity. The empirical discovery that a high masking ratio (~75%) is necessary to overcome spatial redundancy in images is a critical insight that transforms a trivial interpolation task into a challenging semantic reasoning problem. The choice of raw pixel reconstruction (with optional per-patch normalization) over discrete tokenization (e.g., dVAE) is both theoretically grounded and practically advantageous, eliminating the need for a separate tokenizer pre-training stage. The methodology is conceptually simple, mathematically straightforward, and highly scalable, embodying the principle that architectural efficiency and task difficulty are the primary drivers of representation quality.
Results & evaluation
The experimental design is rigorous, comprehensive, and directly addresses the scalability claim. The authors conduct extensive ablations across masking ratios, decoder depth/width, mask token placement, reconstruction targets, augmentation strategies, and sampling patterns, providing clear empirical guidelines for practitioners. The evaluation spans ImageNet-1K pre-training, linear probing, end-to-end fine-tuning, and partial fine-tuning, revealing a crucial and often overlooked insight: linear probing accuracy poorly correlates with downstream transfer performance for masked autoencoders. Transfer evaluations on COCO (detection/segmentation), ADE20K (semantic segmentation), iNaturalist, and Places demonstrate consistent and significant improvements over supervised baselines and competing SSL methods (MoCo v3, BEiT), particularly as model capacity scales. The robustness evaluation on ImageNet-C further validates the generalization strength of the learned features. The results are statistically sound, well-controlled, and effectively isolate the contribution of each design choice.
Reproducibility
Reproducibility is exceptionally high. The implementation avoids specialized sparse operations, relying instead on simple token shuffling/unshuffling and standard Transformer blocks. Training schedules, hyperparameters, data augmentation protocols, and regularization settings are explicitly detailed in the appendices. The authors provide clear recipes for stabilizing large ViT training from scratch, a non-trivial contribution in itself. The simplicity of the pipeline (standard PyTorch/TensorFlow operations, no custom CUDA kernels required) ensures that independent labs can replicate results with minimal friction.
Limitations
The primary limitation lies in the reconstruction objective: predicting raw pixels, while effective, may not optimally capture high-level semantic abstractions compared to token-based or contrastive objectives in certain low-data regimes. The paper acknowledges that linear probing is a poor proxy for representation quality in this context, which complicates rapid benchmarking. Additionally, while the asymmetric design drastically reduces compute, training ViT-Huge still requires substantial resources, and the method's scaling behavior beyond ~1B parameters remains unexplored in this work. The reliance on random masking, while optimal, lacks the structural awareness of object-centric or semantic masking strategies that could further boost sample efficiency.
Broader impact
This work democratizes large-scale vision pre-training by proving that simple, label-free masked modeling can match or exceed supervised pre-training without requiring billions of curated labeled images or complex multi-view contrastive pipelines. It has already catalyzed a paradigm shift toward generative/self-supervised pre-training in vision, influencing architectures across detection, segmentation, and multimodal learning. The authors appropriately note risks related to dataset bias and generative hallucination, which are inherent to any reconstruction-based model trained on web-scale imagery. The method's efficiency and scalability lower the barrier to entry for training foundation models, accelerating research across academia and industry. The paper demonstrates that a simple, asymmetric masked autoencoder with high patch masking ratios is a highly scalable and effective paradigm for self-supervised vision pre-training. By decoupling the heavy encoder from the reconstruction decoder and proving that pixel-level targets suffice for learning transferable representations, the work fundamentally shifts the field away from complex contrastive and tokenization-based pipelines, establishing a new standard for efficient, scalable, and high-performing vision foundation models.
Curator note
He et al.; Meta; high masking ratio MAE; efficient ViT pretraining
Author of this paper? Send a correction.