End-to-End Object Detection with Transformers (DETR)
Abstract
We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at https://github.com/facebookresearch/detr.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper reformulates object detection as a direct set prediction task using Transformers and bipartite matching, eliminating anchors and NMS while establishing a unified, extensible architecture for visual recognition. The work represents a landmark paradigm shift in computer vision, successfully bridging sequence modeling and spatial prediction, and despite initial training inefficiencies and small-object limitations, it spawned an entire research lineage that has since become foundational to modern vision architectures and foundation models.
Methodology
The paper introduces a paradigm shift by reformulating object detection as a direct set prediction problem, eliminating decades of hand-engineered components (anchors, NMS, IoU-based assignment). The core methodological innovation lies in the integration of a Transformer encoder-decoder architecture with a bipartite matching loss (solved via the Hungarian algorithm), which enforces a strict 1-to-1 correspondence between predictions and ground truth objects. The use of a fixed set of learned "object queries" as positional embeddings that attend to the CNN feature map is conceptually elegant and mathematically sound. The training pipeline relies heavily on auxiliary decoding losses across transformer layers and an extended training schedule (500 epochs), which are critical for stabilizing convergence. While the architecture itself is straightforward, the loss formulation and query-based attention mechanism represent a fundamental departure from dense prediction and proposal-based paradigms.
Results & evaluation
Experiments are conducted on the COCO dataset against a heavily optimized Faster R-CNN baseline. DETR achieves comparable overall AP, with notably superior performance on large objects due to the global receptive field of self-attention. However, it underperforms on small objects, a limitation the authors correctly attribute to the lack of multi-scale feature fusion in the initial design. The ablation studies are rigorous, isolating the impact of auxiliary losses, training schedule length, and matching cost weights. The extension to panoptic segmentation demonstrates strong architectural flexibility and outperforms specialized baselines, validating the unified set-prediction philosophy.
Reproducibility
Highly reproducible. The authors explicitly avoid custom CUDA kernels or specialized detection libraries, relying solely on standard ResNet backbones and vanilla Transformer implementations available in mainstream deep learning frameworks. The training code and pretrained models are publicly released, and the methodology is described with sufficient mathematical and architectural detail to allow independent re-implementation. The primary barrier to reproduction is computational: the 500-epoch training schedule demands significant GPU resources, though this is clearly documented.
Limitations
The most prominent limitations are slow convergence (requiring ~500 epochs vs. ~12 for standard detectors), poor small-object detection performance, and high memory/compute overhead during training. The initial Transformer decoder struggles with fine-grained spatial localization, and the fixed number of object queries (typically 100) limits scalability to dense scenes. The paper acknowledges these issues, correctly predicting that subsequent work will address them through architectural refinements (e.g., multi-scale features, deformable attention, denoising training).
Broader impact
DETR catalyzed a fundamental architectural transition in computer vision, proving that sequence-modeling paradigms can effectively replace heavily engineered CNN pipelines for spatial prediction tasks. It laid the groundwork for the entire DETR family (Deformable DETR, DINO, RT-DETR, etc.) and influenced the design of modern vision foundation models that treat detection, segmentation, and tracking as unified set-prediction problems. By removing NMS and anchors, it simplified deployment pipelines and enabled end-to-end differentiable training for complex multi-task vision systems. This paper reformulates object detection as a direct set prediction task using Transformers and bipartite matching, eliminating anchors and NMS while establishing a unified, extensible architecture for visual recognition. The work represents a landmark paradigm shift in computer vision, successfully bridging sequence modeling and spatial prediction, and despite initial training inefficiencies and small-object limitations, it spawned an entire research lineage that has since become foundational to modern vision architectures and foundation models.
Curator note
Carion et al.; detection as set prediction; replaced anchors
Author of this paper? Send a correction.