An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Abstract
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper demonstrates that a pure Transformer applied to image patches can outperform convolutional networks when pre-trained at scale, fundamentally shifting computer vision away from handcrafted inductive biases toward data-driven representation learning. By establishing clear scaling laws, providing rigorous empirical validation across diverse benchmarks, and releasing open-source implementations, this work provided the architectural blueprint for the modern era of foundation models, enabling unprecedented advances in image recognition, multimodal understanding, and generative modeling while setting a new standard for empirical rigor in architecture design.
Methodology
The methodology is elegantly minimal yet conceptually transformative. By partitioning images into fixed-size non-overlapping patches, flattening them, and applying a linear projection to create patch embeddings, the authors bypass all convolutional inductive biases. These embeddings are concatenated with learned positional encodings and processed by a standard Transformer encoder. The core methodological insight is architectural agnosticism: rather than engineering vision-specific attention mechanisms or hybrid CNN-attention blocks, the authors demonstrate that a vanilla Transformer, when scaled appropriately, naturally learns spatial hierarchies and locality from data. This shifts the design philosophy from hardcoding priors to scaling data and compute, establishing a new paradigm where model capacity and dataset size dictate performance ceilings rather than architectural specialization.
Results & evaluation
The experimental framework is exceptionally rigorous and establishes foundational scaling laws for vision models. The authors systematically vary model size, dataset scale (ImageNet-1k, ImageNet-21k, JFT-300M), and patch resolution, revealing a critical crossover point where ViTs surpass state-of-the-art CNNs (ResNet, EfficientNet) in both accuracy and training efficiency. Evaluations span diverse downstream tasks (CIFAR-100, VTAB-1k), demonstrating robust transfer capabilities and strong out-of-distribution generalization. The compute-efficiency analysis is particularly impactful, showing that ViTs achieve superior performance with fewer training FLOPs when pre-trained at scale. Ablation studies on positional embeddings, patch sizes, and data augmentation strategies are thorough and directly inform practical deployment decisions.
Reproducibility
High. The architecture is fully specified with standard components, hyperparameters are clearly documented, and the official codebase was released alongside the paper. While the largest pre-training corpus (JFT-300M) is proprietary, the authors provide comprehensive results on public datasets and the community has extensively reproduced and validated the findings using open alternatives (e.g., DeiT, DINO, MAE). The reliance on standard deep learning frameworks and the absence of custom CUDA kernels or complex training tricks make the methodology highly accessible and straightforward to replicate.
Limitations
The architecture's primary weakness is its heavy dependence on large-scale pre-training; when trained from scratch on small datasets, ViTs underperform CNNs due to the lack of translation equivariance and locality priors, necessitating heavy regularization or knowledge distillation. Additionally, the global self-attention mechanism scales quadratically with sequence length, making direct application to high-resolution images or dense prediction tasks computationally prohibitive without hierarchical or windowed modifications. The model also lacks explicit spatial inductive biases, which can hinder sample efficiency and interpretability in data-constrained regimes.
Broader impact
This work catalyzed a paradigm shift in computer vision, effectively ending the CNN era's architectural dominance and establishing Transformers as the standard backbone for modern vision and multimodal AI. It directly enabled the rapid development of vision-language models, diffusion architectures, and large-scale foundation models that power contemporary AI systems. While the compute requirements for large-scale pre-training raise valid concerns regarding accessibility, energy consumption, and centralization of research, the architectural simplicity and strong scaling properties have democratized high-performance vision research by enabling smaller labs to leverage open pre-trained checkpoints and fine-tune them for specialized applications. The paper demonstrates that a pure Transformer applied to image patches can outperform convolutional networks when pre-trained at scale, fundamentally shifting computer vision away from handcrafted inductive biases toward data-driven representation learning. By establishing clear scaling laws, providing rigorous empirical validation across diverse benchmarks, and releasing open-source implementations, this work provided the architectural blueprint for the modern era of foundation models, enabling unprecedented advances in image recognition, multimodal understanding, and generative modeling while setting a new standard for empirical rigor in architecture design.
Curator note
Dosovitskiy et al.; Transformer for vision; displaced CNN backbones
Author of this paper? Send a correction.