Flamingo: a Visual Language Model for Few-Shot Learning
Abstract
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Flamingo introduces a modular, cross-attention-based architecture that bridges frozen vision and language models to enable robust in-context few-shot learning across interleaved image, video, and text sequences. By demonstrating that a single unified model can outperform specialized fine-tuned baselines across diverse multimodal benchmarks, the paper establishes a foundational paradigm that has since become the standard for modern vision-language model development, fundamentally reshaping how the field approaches multimodal representation learning and adaptation.
Methodology
The paper introduces a highly effective architectural paradigm that bridges frozen, large-scale vision encoders and language decoders via gated cross-attention layers. By keeping the base models frozen and inserting lightweight cross-attention modules, the approach preserves the rich representations learned from massive unimodal pretraining while enabling seamless multimodal fusion. The use of a Perceiver-style resampling mechanism to compress variable-length visual features into a fixed token budget is a critical engineering and algorithmic contribution that makes training computationally feasible. The methodology is elegant, modular, and directly solves the historical bottleneck of joint vision-language training from scratch.
Results & evaluation
The evaluation is exceptionally rigorous, spanning open-ended visual question-answering, image/video captioning, and closed-ended multiple-choice tasks across a wide array of established benchmarks. The few-shot prompting paradigm is thoroughly validated, demonstrating that a single unified model consistently outperforms specialized, fully fine-tuned baselines using only a handful of in-context examples. Comprehensive ablation studies effectively isolate the impact of the cross-attention mechanism, the interleaved web-scale training corpus, and model scaling, providing clear empirical evidence for each design choice.
Reproducibility
While the architectural blueprint and training objectives are clearly specified, exact reproducibility is constrained by the reliance on proprietary, large-scale web corpora and substantial compute infrastructure. However, the core design principles (frozen backbones + cross-attention + interleaved data) have been successfully replicated and adapted in numerous open-source frameworks (e.g., OpenFlamingo, BLIP-2, LLaVA), strongly validating the methodological robustness and transferability of the approach.
Limitations
The model's capabilities are heavily contingent on the scale and quality of the web-scraped pretraining data, which inherently introduces dataset biases, safety risks, and copyright concerns. Inference latency is non-trivial due to the sequential processing of interleaved modalities and the computational overhead of cross-attention over long context windows. Additionally, the work predates the widespread adoption of instruction-tuning and alignment techniques (e.g., RLHF), meaning the model lacks the conversational safety and adherence to complex user constraints seen in later generations.
Broader impact
Flamingo fundamentally shifted the research paradigm from task-specific fine-tuning to unified, in-context multimodal learning. It established the architectural blueprint that underpins the current generation of vision-language models, accelerating progress in embodied AI, robotics, and multimodal assistants. The demonstrated few-shot adaptation capability significantly lowers the barrier to deploying capable models on novel domains without extensive labeled data, though it simultaneously amplifies concerns regarding the environmental footprint of large-scale pretraining and the potential for misuse of highly capable generative systems. Flamingo introduces a modular, cross-attention-based architecture that bridges frozen vision and language models to enable robust in-context few-shot learning across interleaved image, video, and text sequences. By demonstrating that a single unified model can outperform specialized fine-tuned baselines across diverse multimodal benchmarks, the paper establishes a foundational paradigm that has since become the standard for modern vision-language model development, fundamentally reshaping how the field approaches multimodal representation learning and adaptation.
Curator note
Alayrac et al.; DeepMind; few-shot VLM from frozen LLM
Author of this paper? Send a correction.