Adam: A Method for Stochastic Optimization
Abstract
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces the Adam optimizer, a first-order stochastic optimization algorithm that combines bias-corrected adaptive moment estimates with momentum-like updates to achieve efficient, robust, and minimally tuned training. The methodology elegantly bridges adaptive learning rate techniques with classical momentum, supported by rigorous online convex optimization theory and comprehensive empirical validation across diverse architectures, ultimately establishing a new standard for optimization that has been universally adopted across the machine learning community and fundamentally accelerated the development of modern deep learning.
Methodology
The paper introduces Adam (Adaptive Moment Estimation), a first-order optimization algorithm that maintains exponential moving averages of both the gradient (first moment) and squared gradient (second moment). The core methodological innovation lies in the synthesis of adaptive per-parameter learning rates (inspired by AdaGrad/RMSProp) with momentum-like trajectory smoothing, coupled with a mathematically principled bias-correction step that resolves the initialization bias toward zero in early training steps. The derivation is elegant, computationally lightweight (O(d) memory and time per step), and invariant to diagonal gradient rescaling. The theoretical analysis establishes a regret bound under the online convex optimization framework, demonstrating convergence guarantees that match or improve upon contemporaneous adaptive methods. While the convexity assumptions in the proof do not strictly hold for modern deep networks, the algorithmic design is remarkably robust to non-convex, high-dimensional, and noisy gradient landscapes.
Results & evaluation
The empirical evaluation is thorough and well-structured, covering logistic regression, CNNs on MNIST/CIFAR-10, RNNs on IMDB sentiment and character-level language modeling, and autoencoders. Adam consistently matches or outperforms strong baselines (SGD with momentum, Nesterov, AdaGrad, AdaDelta, RMSProp), demonstrating faster convergence, greater stability, and reduced sensitivity to hyperparameter tuning. The experiments use standard datasets and report both training and validation metrics, providing clear evidence of practical superiority. While subsequent research has revealed nuanced generalization trade-offs between adaptive methods and SGD in certain regimes, the paper's empirical claims are rigorously supported within the evaluated scope and effectively demonstrate the algorithm's broad applicability.
Reproducibility
The algorithm is specified with complete mathematical precision, including explicit update equations, default hyperparameters (α=0.001, β1=0.9, β2=0.999, ε=1e-8), and initialization procedures. The provided pseudocode is concise and directly translatable to implementation. No proprietary data, specialized hardware, or complex infrastructure is required, making the method trivially reproducible. Its simplicity has led to native integration in all major deep learning frameworks, ensuring perfect reproducibility and zero implementation friction.
Limitations
The theoretical analysis relies on convexity and bounded gradient assumptions, which limit its direct applicability to the highly non-convex loss surfaces of deep neural networks. Subsequent work has identified convergence pathologies in certain online settings (later addressed by AMSGrad) and demonstrated that Adam's adaptive scaling can occasionally yield flatter minima with marginally worse generalization than carefully tuned SGD. The algorithm also doubles the memory footprint per parameter compared to vanilla SGD due to storing two moment vectors, which can become a bottleneck in extreme-scale training. Finally, the default ε value, while empirically robust, can occasionally cause numerical instability in low-precision training regimes.
Broader impact
Adam fundamentally transformed the standard practice of neural network training, becoming the de facto optimizer across computer vision, NLP, reinforcement learning, and generative modeling for nearly a decade. Its robustness and minimal tuning requirements dramatically lowered the barrier to training complex architectures, accelerating both academic research and industrial deployment. The algorithm's design principles catalyzed an entire subfield of adaptive optimization research, inspiring numerous variants (AdamW, Lion, Sophia, etc.) and continuing to shape how practitioners approach large-scale stochastic optimization. By enabling faster, more stable training with intuitive defaults, Adam has had a profound, field-wide impact on the scalability and accessibility of machine learning. This paper introduces the Adam optimizer, a first-order stochastic optimization algorithm that combines bias-corrected adaptive moment estimates with momentum-like updates to achieve efficient, robust, and minimally tuned training. The methodology elegantly bridges adaptive learning rate techniques with classical momentum, supported by rigorous online convex optimization theory and comprehensive empirical validation across diverse architectures, ultimately establishing a new standard for optimization that has been universally adopted across the machine learning community and fundamentally accelerated the development of modern deep learning.
Curator note
Default optimizer for most modern ML; Kingma & Ba
Author of this paper? Send a correction.