Segment Anything
Abstract
We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive -- often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at https://segment-anything.com to foster research into foundation models for computer vision.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces a promptable segmentation foundation model, a novel iterative data engine, and the largest segmentation dataset to date, establishing a scalable paradigm for zero-shot dense prediction. The work successfully bridges the gap between interactive segmentation and foundation models through a deliberately simple, highly efficient architecture and a meticulously engineered data collection pipeline. By rigorously validating zero-shot transfer across 23 diverse datasets and demonstrating strong composability in downstream tasks, the authors provide compelling empirical evidence that scale and prompt engineering can generalize dense prediction capabilities beyond training distributions. The open release of SAM and SA-1B will serve as a foundational resource for the vision community, accelerating research in modular AI systems and setting a new standard for data-driven foundation model development.
Methodology
The paper introduces a cohesive, systems-level methodology comprising a novel promptable segmentation task, a streamlined transformer-based architecture (SAM), and an iterative model-in-the-loop data engine. While the core architectural components (ViT image encoder, positional prompt encoding, lightweight mask decoder) are not radically novel in isolation, their integration is highly deliberate, prioritizing amortized efficiency, real-time interactivity, and ambiguity resolution. The most significant methodological contribution is the three-stage data engine (assisted-manual → semi-automatic → fully automatic), which successfully scales annotation to 1.1B masks by leveraging the model's own improving capabilities. The ambiguity-aware design, which predicts multiple masks per prompt and ranks them via confidence scores, elegantly addresses a long-standing limitation in interactive segmentation. The approach trades architectural complexity for scale and composability, a pragmatic and well-justified design choice for foundation model development.
Results & evaluation
The experimental evaluation is extensive, rigorous, and thoughtfully designed to validate zero-shot generalization. The authors compile a diverse suite of 23 datasets spanning medical, autonomous driving, robotics, and natural imagery, demonstrating robust cross-domain transfer. Recognizing that standard mIoU penalizes valid but ambiguous predictions, the authors supplement automatic metrics with human quality ratings, providing a more nuanced performance picture. Downstream evaluations (edge detection, object proposals, instance segmentation, text-to-mask) effectively demonstrate composability and prompt engineering capabilities. Ablation studies systematically isolate the contributions of data engine stages, dataset scale, and backbone capacity. While SAM does not universally surpass fully supervised SOTA (as expected for a zero-shot baseline), the results consistently show competitive performance and reveal valuable empirical insights about scale, ambiguity, and dataset biases.
Reproducibility
Excellent. The authors release the full SA-1B dataset (11M images, 1.1B masks), model weights, and training code under a permissive Apache 2.0 license. The paper provides detailed architectural specifications, training hyperparameters, loss formulations, and data engine protocols. The inclusion of a publicly accessible web demo and comprehensive dataset/model cards further lowers the barrier to replication and downstream adaptation. The methodological transparency and open release set a high standard for foundation model research.
Limitations
The authors candidly acknowledge several limitations: SAM struggles with fine-grained structures and small disconnected components, occasionally hallucinates artifacts, and does not produce boundaries as crisp as specialized zoom-in methods. The heavy image encoder prevents true real-time inference, and the model is not optimized for high-IoU interactive refinement with many user clicks. Semantic and panoptic segmentation cannot be achieved through simple prompting alone, requiring external components. The text-to-mask capability remains preliminary and relies on CLIP embedding alignment rather than direct text supervision. Additionally, while demographic fairness is largely consistent, bias emerges in clothing segmentation, highlighting downstream composition risks. These limitations are well-documented and do not undermine the core contribution.
Broader impact
This work fundamentally shifts the paradigm for dense prediction in computer vision, establishing promptable segmentation as a viable foundation model task. The unprecedented scale of SA-1B and the composability of SAM will serve as a critical infrastructure layer for research and applications across medical imaging, robotics, AR/VR, and content creation. By demonstrating that a model-in-the-loop data engine can overcome the annotation bottleneck for dense tasks, the paper provides a scalable blueprint for future vision foundation models. The Responsible AI analysis is a positive step, though the massive compute requirements and data licensing considerations warrant ongoing community scrutiny. Overall, the release will catalyze widespread adoption, modular system design, and new research directions in scalable vision. This paper introduces a promptable segmentation foundation model, a novel iterative data engine, and the largest segmentation dataset to date, establishing a scalable paradigm for zero-shot dense prediction. The work successfully bridges the gap between interactive segmentation and foundation models through a deliberately simple, highly efficient architecture and a meticulously engineered data collection pipeline. By rigorously validating zero-shot transfer across 23 diverse datasets and demonstrating strong composability in downstream tasks, the authors provide compelling empirical evidence that scale and prompt engineering can generalize dense prediction capabilities beyond training distributions. The open release of SAM and SA-1B will serve as a foundational resource for the vision community, accelerating research in modular AI systems and setting a new standard for data-driven foundation model development.
Curator note
Kirillov et al.; Meta; promptable segmentation; billion-mask dataset
Discussion
Talks & videos
Author of this paper? Send a correction.