A Distribution-Free Separation Between Learned and Fixed Features
Abstract
The success of neural networks is largely attributed to their ability to learn useful representations from training data. However, known results that prove a computational advantage of learned features over fixed ones are restricted to specific input distributions. We settle the COLT open question of Feldman, Kamath, and Srebro (2026) by showing that this advantage can persist in the distribution-free setting. We construct a family of hypothesis classes that can be efficiently learned by a natural gradient-based algorithm on a two-layer ReLU network, but cannot be represented by linear predictors over any fixed feature map with polynomial dimension (and runtime). Specifically, for any sufficiently small error and any sufficiently large runtime budget R (depending on the error), there exists a hypothesis class that the neural network learns under any input distribution to the desired expected classification error in time at most R, while the dimension complexity (governing the dimension needed by linear predictors) is superpolynomial in R.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Resolves a COLT open question by proving a distribution-free separation between learned and fixed features. The paper provides a rigorous theoretical foundation for the computational advantage of neural network feature learning, demonstrating that certain hypothesis classes can be learned efficiently by gradient-based methods on ReLU networks despite having superpolynomial dimension complexity for fixed feature maps.
Methodology
The paper proposes a rigorous theoretical framework to resolve an open question regarding the computational advantage of learned features over fixed features in a distribution-free setting. The methodology involves constructing a specific family of hypothesis classes (disjunctions of conjunctions on Boolean inputs) and demonstrating that a natural two-stage gradient-based algorithm (signSGD on hidden layer, projected SGD on output layer) for a two-layer ReLU network can learn these classes efficiently. The core technical contribution is a proof that while the dimension complexity of these classes is superpolynomial in the runtime budget, the neural network learns them in polynomial time. The proof leverages polynomial approximation techniques, Gaussian noise injection to create "useful" neurons that act as counters, and Fourier analysis on the Boolean cube to show that linear combinations of trained neurons can approximate the target functions. The approach is mathematically sophisticated, combining elements of learning theory, harmonic analysis, and optimization.
Results & evaluation
This is a purely theoretical paper. There are no empirical experiments, datasets, or benchmarks. The "evaluation" consists of rigorous mathematical proofs establishing the existence of the separation and the correctness of the learning algorithm's runtime and error bounds. The lack of empirical validation is expected for this subcategory but limits immediate practical applicability.
Reproducibility
The paper provides a detailed algorithm (Algorithm 1) and specifies hyperparameters in a table. The proofs are self-contained with appendices detailing the derivation of key lemmas. However, without code or empirical results, "reproducibility" in the standard experimental sense is not applicable. The theoretical claims are reproducible in the sense that a mathematician can verify the proofs.
Limitations
The results are restricted to a specific class of functions (DNF-like structures on Boolean inputs) and a specific network architecture (two-layer ReLU). The learning algorithm relies on injected Gaussian noise in the first stage, which is not standard in practical deep learning training. The separation is established for "natural" gradient-based algorithms, but the definition of "natural" is somewhat heuristic. The results do not directly translate to modern deep learning practices involving many layers, batch normalization, or complex optimizers.
Broader impact
This paper contributes to the foundational understanding of why neural networks are powerful. By showing that feature learning provides a computational advantage even in the worst-case (distribution-free) setting, it strengthens the theoretical case for representation learning. It may influence future theoretical work on the limits of linear models and the necessity of non-linear feature learning. However, its direct impact on engineering practice is limited due to the theoretical nature and specific assumptions. Resolves a COLT open question by proving a distribution-free separation between learned and fixed features. The paper provides a rigorous theoretical foundation for the computational advantage of neural network feature learning, demonstrating that certain hypothesis classes can be learned efficiently by gradient-based methods on ReLU networks despite having superpolynomial dimension complexity for fixed feature maps.
Author of this paper? Send a correction.