Infinite ResNets are Gaussian Mean ODEs
Abstract
Understanding the training dynamics of deep neural networks in large-scale limits is a question both of foundational and of practical importance. Prior works have obtained such descriptions through dynamical mean field theory (DMFT), yielding a system of non-Markovian dynamics that are challenging to analyze and that obscure the optimization structure of the original dynamics. In this paper, we obtain instead a simple description of the large-scale limit training dynamics for ResNets in standard shape scalings and in the $1/\text{depth}$ branch scaling that allows maximal (local) feature updates. For a model trained with gradient descent (GD) from a random initialization, the limit dynamics is simply GD on a Neural Mean ODE with Gaussian initial weights. Our analysis covers population or empirical risk GD in the joint limit of large width, depth and embedding, input, and output dimensions. We obtain approximation scaling laws that depend on the shape of the ResNet and the covariance spectrum of the data. We also illustrate the flexibility of our approach by discussing Mixture of Expert models, where likewise the limit admits an intuitive (nested) Mean ODE description.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper establishes that the large-scale training dynamics of ResNets converge to Gradient Descent on a Gaussian Mean ODE, providing a tractable and structurally faithful limit model. This represents a significant advance in the theoretical analysis of deep learning by simplifying the complex non-Markovian dynamics of DMFT into a standard optimization problem, enabling sharper convergence rates and deeper insights into the role of architecture shape and data covariance.
Methodology
The paper proposes a rigorous theoretical framework for analyzing the training dynamics of Residual Neural Networks (ResNets) in the joint limit of large depth, width, and embedding dimension. The core innovation is deriving a "Gaussian Mean ODE" limit, which simplifies the complex non-Markovian dynamics typically found in Dynamical Mean Field Theory (DMFT) into a standard Gradient Descent (GD) process on a Neural Mean ODE. The methodology relies on a two-stage limit: first, depth divergence leads to a Mean ODE, and second, embedding dimension divergence with Gaussian initialization leads to a dimension-free Gaussian Mean ODE. The authors provide quantitative convergence rates and prove that the limit dynamics preserve the optimization structure of the original network, a significant improvement over prior DMFT approaches that obscure this structure. The use of Hilbert space formalism allows for generalization to infinite-dimensional input/output spaces.
Results & evaluation
As a theoretical paper, experimental evaluation is limited to numerical validations of the derived convergence rates and the behavior of the limit model. The paper includes plots demonstrating that the empirical errors match the theoretical bounds (e.g., $O(1/L + D/ML)$) and illustrates the "quasi-smoothness" of the limit objective. While not providing large-scale empirical benchmarks, the numerical experiments effectively validate the theoretical claims and the tightness of the derived bounds.
Reproducibility
The paper is highly reproducible in a theoretical sense, with clear definitions, lemmas, and theorems. The code for numerical experiments is likely available (though not explicitly linked in the provided text snippet, standard practice for such papers from this group implies code availability). The mathematical derivations are detailed, allowing other researchers to verify the proofs. The reliance on AI for specific proof steps (GPT-6 Astra) is disclosed, which is a novel aspect of reproducibility/transparency in the research process itself.
Limitations
The analysis is restricted to ResNets with two-layer perceptron blocks and specific scaling regimes (MLU/1-depth). It does not cover attention mechanisms or more complex architectures directly, although it mentions MoE. The results are for Gradient Descent; extensions to SGD or other optimizers are not covered. The assumption of Gaussian initialization for the sharpest results is a limitation, though they do discuss non-Gaussian cases with weaker bounds.
Broader impact
This work provides a crucial bridge between the practical architecture of ResNets and the theoretical understanding of their training dynamics. By showing that the limit is simply GD on a Mean ODE, it makes the analysis of deep network training significantly more tractable. This could lead to better hyperparameter tuning strategies, understanding of generalization in deep networks, and potentially new optimization algorithms designed specifically for the Mean ODE limit. It contributes to the foundational understanding of why deep networks train well. The paper establishes that the large-scale training dynamics of ResNets converge to Gradient Descent on a Gaussian Mean ODE, providing a tractable and structurally faithful limit model. This represents a significant advance in the theoretical analysis of deep learning by simplifying the complex non-Markovian dynamics of DMFT into a standard optimization problem, enabling sharper convergence rates and deeper insights into the role of architecture shape and data covariance.
Author of this paper? Send a correction.