Learning Transferable Visual Models From Natural Language Supervision
Abstract
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces CLIP, a contrastive vision-language pre-training framework that learns transferable image representations from 400 million internet-sourced image-text pairs, enabling robust zero-shot transfer across diverse vision tasks and fundamentally shifting the field toward language-supervised, open-vocabulary perception. The work's methodological simplicity, combined with rigorous scaling analysis, comprehensive cross-dataset benchmarking, and profound empirical findings on robustness and data efficiency, establishes it as a paradigm-defining contribution that has permanently altered research trajectories in computer vision, multimodal learning, and foundation models.
Methodology
The paper introduces CLIP (Contrastive Language-Image Pre-training), a conceptually elegant framework that jointly trains an image encoder and a text transformer to maximize cosine similarity between correctly paired image-text samples while minimizing it for all other pairings within a large batch. By replacing autoregressive or masked language modeling objectives with a symmetric cross-entropy contrastive loss, the authors achieve a 4x training efficiency gain over predictive baselines. The methodology deliberately avoids complex architectural innovations, instead relying on scale (400M web-scraped pairs), careful engineering (attention pooling, learnable temperature, linear projections), and a novel inference-time mechanism where natural language prompts dynamically synthesize zero-shot linear classifiers. The approach effectively reframes visual recognition as a retrieval problem in a shared multimodal embedding space.
Results & evaluation
The experimental design is exceptionally thorough and sets a new standard for evaluating representation learning. The authors benchmark zero-shot transfer across 30+ datasets spanning fine-grained classification, OCR, action recognition, and geo-localization, demonstrating that CLIP matches or exceeds fully supervised baselines on many tasks without any task-specific training. The paper rigorously documents scaling laws, showing predictable log-linear improvements in zero-shot accuracy as compute increases across 44x. Linear probe evaluations confirm that CLIP learns broader, more transferable features than ImageNet-supervised models, particularly excelling on out-of-domain tasks. The robustness analysis across 7 natural distribution shifts is particularly compelling, revealing that zero-shot CLIP reduces the performance degradation gap by up to 75% compared to supervised counterparts.
Reproducibility
High. The authors provide exhaustive training details including optimizer settings, learning rate schedules, batch sizes (32,768), mixed-precision techniques, gradient checkpointing, and sharded similarity computation. The dataset construction pipeline (WIT) is transparently described, and both code and pre-trained weights are publicly released. While the compute requirements (up to 592 V100 GPUs for 18 days) limit exact replication for most academic groups, the methodological transparency and open-sourced artifacts ensure strong reproducibility for well-resourced labs and facilitate widespread adoption.
Limitations
Zero-shot CLIP exhibits notable weaknesses on highly specialized, abstract, or reasoning-heavy tasks (e.g., medical pathology, satellite imagery, synthetic counting), where performance lags significantly behind supervised models. The approach is highly sensitive to prompt engineering, introducing a manual, dataset-specific tuning overhead that partially undermines the "zero-shot" ideal. The model inherits and amplifies societal biases present in internet-scale text, raising fairness and safety concerns. Additionally, the pre-training compute barrier is substantial, and the paper acknowledges that zero-shot accuracy still trails fully supervised fine-tuning by 10-20% on many benchmarks.
Broader impact
CLIP fundamentally redefined computer vision by demonstrating that noisy, web-scale natural language supervision can outperform meticulously curated labeled datasets, catalyzing the shift toward open-vocabulary, language-guided perception. It directly enabled the generative AI revolution, serving as the foundational vision-language backbone for DALL-E, Stable Diffusion, and countless downstream multimodal systems. The work also established zero-shot evaluation as a critical metric for assessing true model generalization and robustness. However, it simultaneously highlights pressing ethical and environmental challenges, including copyright concerns around web-scraped data, bias propagation, and the carbon footprint of massive-scale pre-training. This paper introduces CLIP, a contrastive vision-language pre-training framework that learns transferable image representations from 400 million internet-sourced image-text pairs, enabling robust zero-shot transfer across diverse vision tasks and fundamentally shifting the field toward language-supervised, open-vocabulary perception. The work's methodological simplicity, combined with rigorous scaling analysis, comprehensive cross-dataset benchmarking, and profound empirical findings on robustness and data efficiency, establishes it as a paradigm-defining contribution that has permanently altered research trajectories in computer vision, multimodal learning, and foundation models.
Curator note
Radford et al.; zero-shot transfer; most influential vision-language model
Discussion
Talks & videos
Related later work
Author of this paper? Send a correction.