Attention Is All You Need
Abstract
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The Transformer architecture replaces recurrent and convolutional sequence modeling with a purely attention-based design, achieving superior accuracy and training efficiency. By demonstrating that self-attention alone can capture complex sequential dependencies while enabling massive parallelization, this work fundamentally redefined the trajectory of deep learning, serving as the foundational architecture for virtually all modern foundation models and establishing a new paradigm for scalable, cross-domain representation learning.
Methodology
The paper introduces a paradigm-shifting architecture that entirely replaces recurrent and convolutional layers with multi-head self-attention mechanisms. The methodological design is exceptionally clean and mathematically grounded: scaled dot-product attention stabilizes gradient flow, sinusoidal positional encodings inject sequence order without recurrence, and residual connections paired with layer normalization enable stable training of deep stacks. The architectural choices are rigorously motivated through systematic ablation studies, demonstrating that attention alone can capture both local and global dependencies more effectively than RNNs/CNNs while enabling massive parallelization across sequence dimensions.
Results & evaluation
Experiments are conducted on standard WMT 2014 English-to-German and English-to-French translation benchmarks, achieving state-of-the-art BLEU scores of 28.4 and 41.8, respectively. Crucially, the model achieves these results with a fraction of the training compute (3.5 days on 8 GPUs) compared to prior ensemble-based RNN/CNN approaches that required weeks or months. The evaluation extends to English constituency parsing, demonstrating strong cross-task generalization even under data-constrained regimes. The empirical results are robust, well-controlled, and include comprehensive ablations on attention head count, key/query/value dimensions, and positional encoding variants.
Reproducibility
Excellent. The authors provide exhaustive architectural specifications, hyperparameter schedules, optimizer settings, and training details. The release of the tensor2tensor library ensures that the community can immediately reproduce, extend, and benchmark the architecture. The clear mathematical formulation and open-source implementation set a new standard for reproducibility in deep learning research.
Limitations
The self-attention mechanism exhibits quadratic time and memory complexity with respect to sequence length ($O(N^2)$), which initially constrained its application to very long sequences. Additionally, autoregressive decoding remains inherently sequential, limiting generation throughput. The authors acknowledge these constraints and explicitly propose future work on local/restricted attention and parallel generation strategies, which subsequent research has extensively addressed.
Broader impact
The Transformer fundamentally redefined sequence modeling and catalyzed the modern era of large-scale AI. Its architecture became the foundational backbone for BERT, GPT, T5, Vision Transformers, and multimodal foundation models, driving unprecedented advances across NLP, computer vision, audio processing, and scientific computing. The work democratized high-performance sequence modeling by drastically reducing training costs and enabling scalable, parallelizable architectures that continue to dominate both academic research and industrial deployment. The Transformer architecture replaces recurrent and convolutional sequence modeling with a purely attention-based design, achieving superior accuracy and training efficiency. By demonstrating that self-attention alone can capture complex sequential dependencies while enabling massive parallelization, this work fundamentally redefined the trajectory of deep learning, serving as the foundational architecture for virtually all modern foundation models and establishing a new paradigm for scalable, cross-domain representation learning.
Curator note
Vaswani et al.; most cited ML paper ever; foundation of modern AI
Discussion
Talks & videos
Related later work
- BERT 2018
- GPT-2 / GPT-3 2020
- T5 2019
- Vision Transformer (ViT) 2020
- DALL-E / GPT image models 2021
- Whisper 2022
- AlphaFold 2 2021
Author of this paper? Send a correction.