The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Abstract
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper identifies sparse, trainable subnetworks within dense randomly-initialized networks by isolating initialization as the critical factor for trainability, fundamentally reshaping research in model compression, optimization landscapes, and efficient deep learning. This work represents a conceptual breakthrough that reframes pruning from a post-hoc compression technique to an initialization discovery mechanism, spawning a highly active subfield and providing a rigorous empirical foundation for understanding why overparameterized networks generalize effectively despite their size.
Methodology
The paper introduces iterative magnitude pruning combined with weight rewinding to the original random initialization, a deceptively simple procedure that fundamentally decouples architectural sparsity from initialization quality. The core methodological contribution is the isolation of initialization as the critical variable for trainability in sparse subnetworks. The experimental design systematically controls for confounding factors (e.g., learning rate schedules, weight magnitude distributions, and random pruning baselines) to validate that the observed performance stems from fortuitous initial weight configurations rather than inductive biases of the pruning process itself. The formulation of the "lottery ticket hypothesis" provides a clear, testable theoretical framing that bridges empirical pruning observations with optimization landscape analysis.
Results & evaluation
Experiments are conducted on standard benchmarks (MNIST, CIFAR-10) using fully-connected and convolutional architectures (VGG, ResNet). The results robustly demonstrate that subnetworks comprising 10-20% of the original parameters can match or exceed the original network's test accuracy when trained from their original initialization, while training from scratch or from random initialization fails. The empirical curves showing faster convergence and higher final accuracy for winning tickets are compelling. However, the evaluation is constrained to small-scale vision datasets and standard feed-forward/convolutional architectures; the methodology's scalability to large-scale datasets (e.g., ImageNet) or modern architectures (Transformers) is not addressed in this initial work, though subsequent literature has extensively validated and extended it.
Reproducibility
The algorithm is explicitly defined with clear hyperparameters, pruning schedules, and initialization protocols. The weight-rewinding mechanism is straightforward to implement in any standard deep learning framework. The authors released their code, and the methodology has been independently reproduced and extended by dozens of research groups with high fidelity. The experimental setup is transparent, and ablation studies (e.g., one-shot vs. iterative pruning, random sparsity baselines) provide sufficient detail for replication.
Limitations
The primary limitation is the lack of training-time efficiency: identifying a winning ticket requires pre-training the dense network, which negates immediate computational savings during the search phase. Additionally, the hypothesis is empirically validated only on relatively small datasets and standard CNNs; it does not initially address dynamic sparsity, hardware-aware deployment, or the role of batch normalization in masking initialization effects. The theoretical underpinnings remain largely empirical, with no formal convergence guarantees or landscape analysis provided in the original submission.
Broader impact
This work catalyzed a paradigm shift in how the community views overparameterization, initialization, and model compression. It directly inspired the fields of sparse training, early-bird tickets, and initialization-aware pruning, influencing both algorithmic research and hardware-efficient AI deployment. By demonstrating that effective subnetworks exist at initialization, it challenges the necessity of dense training and opens pathways for training large models with a fraction of the compute, with significant implications for sustainable AI and democratized access to large-scale model training. The paper identifies sparse, trainable subnetworks within dense randomly-initialized networks by isolating initialization as the critical factor for trainability, fundamentally reshaping research in model compression, optimization landscapes, and efficient deep learning. This work represents a conceptual breakthrough that reframes pruning from a post-hoc compression technique to an initialization discovery mechanism, spawning a highly active subfield and providing a rigorous empirical foundation for understanding why overparameterized networks generalize effectively despite their size.
Curator note
Frankle & Carlin; sparse subnetworks; influential pruning theory
Author of this paper? Send a correction.