Sequence to Sequence Learning with Neural Networks
Abstract
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Introduces the foundational LSTM-based encoder-decoder architecture for sequence-to-sequence learning, demonstrating that a simple, end-to-end neural approach can outperform complex statistical machine translation systems and establishing the paradigm that would dominate NLP for years. The paper's methodological clarity, rigorous empirical validation on WMT benchmarks, and the pivotal discovery of source-sequence reversal for optimization stability collectively represent a landmark contribution that not only solved a major applied problem but also defined the architectural trajectory of modern generative AI.
Methodology
The paper introduces a clean, end-to-end encoder-decoder architecture using two stacked LSTMs. The encoder maps a variable-length source sequence to a fixed-dimensional context vector, while the decoder autoregressively generates the target sequence conditioned on this vector. The most significant methodological contribution is the empirical discovery that reversing the order of the source sequence dramatically improves optimization by creating short-term dependencies between the beginning of the source and target sequences, mitigating vanishing gradient issues in early training steps. The approach deliberately strips away the complex alignment, reordering, and feature engineering pipelines of statistical machine translation (SMT), relying purely on the representational capacity of deep recurrent networks. While conceptually simple, the design choices are highly principled and mathematically sound.
Results & evaluation
Rigorously evaluated on the WMT'14 English-French dataset, a standard and highly competitive benchmark. The model achieves a standalone BLEU score of 34.8, surpassing a strong phrase-based SMT baseline (33.3). When used to rerank 1,000 SMT n-best hypotheses, performance jumps to 36.5, demonstrating strong complementary learning. The paper thoroughly investigates sentence length robustness, showing the model maintains performance on long sequences where traditional RNNs typically fail. Qualitative analysis confirms the model learns syntactically coherent phrase structures and exhibits invariance to active/passive transformations. The experimental setup uses standard metrics, clear baselines, and ablation-style insights (e.g., the reversal trick), making the results highly credible.
Reproducibility
The architecture, layer dimensions, optimization procedure (SGD with gradient clipping, learning rate scheduling), and data preprocessing steps are clearly documented. Although published before the modern era of mandatory code release, the mathematical formulation and hyperparameter specifications are sufficiently detailed for independent reproduction. The use of publicly available WMT datasets ensures that any researcher can replicate the experimental pipeline. Modern open-source implementations (e.g., in TensorFlow/PyTorch) have since validated the reproducibility of the core results.
Limitations
The primary architectural limitation is the fixed-dimensional context vector, which creates a severe information bottleneck for long sequences and forces the encoder to compress all source information into a single static representation. This bottleneck inherently limits translation quality for very long documents and motivated the subsequent development of attention mechanisms. Additionally, the model lacks explicit word alignment, making it difficult to interpret internal representations or enforce hard constraints (e.g., terminology consistency). Training requires substantial computational resources and large parallel corpora, limiting applicability to low-resource language pairs without additional techniques.
Broader impact
This work catalyzed the paradigm shift from statistical to neural machine translation, directly influencing industry deployment of translation systems and rendering decades of SMT engineering obsolete. The seq2seq framework generalized rapidly beyond translation to text summarization, conversational AI, speech recognition, and program synthesis. Crucially, the identified bottleneck in the fixed context vector directly inspired the attention mechanism, which in turn led to the Transformer architecture. The paper established the blueprint for autoregressive sequence generation that underpins modern large language models. Introduces the foundational LSTM-based encoder-decoder architecture for sequence-to-sequence learning, demonstrating that a simple, end-to-end neural approach can outperform complex statistical machine translation systems and establishing the paradigm that would dominate NLP for years. The paper's methodological clarity, rigorous empirical validation on WMT benchmarks, and the pivotal discovery of source-sequence reversal for optimization stability collectively represent a landmark contribution that not only solved a major applied problem but also defined the architectural trajectory of modern generative AI.
Curator note
Sutskever et al.; foundation of seq2seq NMT
Author of this paper? Send a correction.